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ABSTRACT 

Chanute Air Force Base was the first military 
training center to undertake an extensive investigation of the use of 
the PLATO IV ^system in technical training. The service test was to 
compare a conventionally taught course in general vehicle maintenance 
with a course taught with computer-based instructional materials. 
Part I of this document is the final report of the project and 
discusses site history and management efforts. Topics include the 
research agreementr selection and training of PLATO authorsr 
evolution of the project management, lesson development procedures, 
and Instructional Systems Development (ISD) management of the service 
^ test. Written by personnel from the Computer-based Education Research 
NLaboratory of the University of Illinois, iho acted as evaluation, 
instructional, and programming consultants for the service test. Part 
II is a chapter- by-chapter critique of the final report. 
(Author/JAB) 
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i . 
FOREWORD 

The School of Applied Aerospace. Sciences at Chanute AFB, 
Illinoi^, was the first military draining center to undertake' - 
an extensive investigation of the use of the PLATO IV system 
in technical training. This research was sponsored by the 
Advanced Research Projects Agency (ARPAT of the Department of 
DeT¥nse~andnth¥~Air Train ing Command (ATC) of the United 
States Air Force. ARPA provided hardware and hardware sup- 
port while ATC staffed the project. 

• It was originally intended that the project have 25 
PLATO IV terminals; However, by the project's- end 30 term- . 
inals were, and still are, in place at Chanute. These extra 
terminals were necessitated by the practical demands of using 
the PLATO system as a'n operational medium of instr-jct ion . 
Each of these terlhinals is equipped with a graphic display 
panel, touch panel, and microfiche slide selector. 

- Historical Summary 

ARPA funding began in July, 1972, and ceased in June, 
1976. During this time, the service test went through four 
distinct phases. Each phase was characterized by distinct 
tnodes of management and objective3. 

Phase one . The initial phase of the service test was 
characterized by uncertainty in leadership, objectives and 
orientation. The leadership and objectives both underwent 
several changes during this phase. LikevJise, the orientation 
of the project was initially towards producing courseware 
for a large scale test -of computer-based technical training. 
After some time. this was given up and interest: shifted from 
a primarily operational emphasis to one of research. This 
vacillation of the project was probably a result of lack of 
active interest on the part of the responsible Air Force 
agencies. 
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Phase two . When a more active interest in the service 
.test was taken, the projec„t was put in the hands of an exper- 
ienced curriculum designer and was given specific objectives 
During this phase, CBE materials were to be produced for use 
<>in operational training. Any research objective became in- 
cidental to that, of producing courseware for Air Force train- 
ees. By the modification of some materials from the first 
phase and creating the rest, this objective was met. The 
use of the resulting CBE materials byAir Force trainees has 
resulted in a rich source of data concerning the use of CBE 
ir. a military training environment. This phase came to an 
end in-June, 1975. 

Phase three. Following the validation and implementa- 
tion of these .materials, the service test undertook some 
small researches and' courseware development efforts. An 
evaluation of this latter part is found in Chapter IX of "~ 
Dallman et al .(1977) . 

Phase four . When ARPA fundirig stopped in June, 1976, 
ATC funded a brief follow-on study of three months. During 
this period information was to be gathered concerning spe- 

use. These studies, focused on the 
cost and times savings due to PLATO us?, the ef f i6 iency^^a^- 
PLATO-based testing, and the volume of trainess that could be 
trained with the PLATO courseware developed during the first 
two phases. "^This study was completed in October, 1976. The 
three studies of this phase .-.re reported in Main ( 1 977). 

The information gathered in the fourth phase provided a 
basis for the Chanute Center Commander to request funding 
from ATC f or^ the use of PLATO as an operational training med- 
ium at Chanute AFB. This funding was provided so tha^: the 
30 PLATC terminals presently located at Chanute are currently 
delivering computer-based technical training to Air Force 
trainees. This training largely consists of PLATO lessons 
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developed during the first two phases of the* service test. 
They are now being used at a approximate rate of 7Q00 stud- 
ent contact hours per year with an estimated yearly flow of 
350 trainees. 

This volume is concerned almost exclusively with the 
first two phases of the service te^t. This period of the 
service test accounted for almost all of the CBE courseware 
- and most of its data on the use of CBE in a military training 
environment. In addition to a wealth of data,- these phases 
present some very valuable lessons in the management and 
admin ist);;^tion of such a service test. Because of the 
importance of these lessons and data and because the later 
phases of the service test did not add^'substantially to what 
was already known, it seemed appropri'ate to focus attention 
on^the first two phases. 

The Critique and Summary of the Service Test 

Part One . Part One gives a fairly detailed summary of 
the history of the service test's first two phase. During 
the first phase, the service test floundered. The sources of 
the service test difficulties both at the site of the test 
and from ATC and AFHRL would be evident to any casual obser- 
ver of the service test. Neverthless, in such matters it is 
always good to have one's intuitions about a situation con- 
firmed. The second phase of the service test serves this- 
purpose. At the beginning of that ph-ase, most of the appar- 
ent shortcomings of the first phase were eliminated. The 
outcome of the second phase has already been mentioned. 

Part Two . Part Two is a commentary on the Evaluation 
of PLATO IV in Vehicle Maintenance Training (Dallman et al . , 
1977). Part Two is intended to be read in this evaluation 
report. Its Foreword (p. 62) descr ibes' its purpose and scope. 
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ESTABLISHMENT OF THE AIR FORCE SERVICE TEST 



Research Agreement 

The-^first research agreement that governed the United 
States Air Force's service test of ^the PLATO IV system was 
signed by the involved agencies in July, 1972. Thfe pro- 
jected three year service' test was, in the words of the 
agreement, to determine the "cost effectiveness, acceptance, 
and incorporabili'ty (Into on-going training) and reiiabil- 
ity"> of computer-based education ( CBE ) in a military train- 
ing environment.^ The research agreement also outlined the 
responsibilities of the agencies that would be involved in 
the« service test. These agencies we,re the Air Training 
Comrhan4" ( ATC) , the Air Force Systems Command Human Rasources 
Laboratory (AFHRL),-the Advance'd Research Projects Agency 
(ARPA) of the Department of Defense, and the Computer-based 
Education Research Laboratory '( CERL) of the University of 
Illinois. The following summary of the responsibilities of 
each of these organizations is abstracted from this research 
agreeement . 

The ATC School of Applied Aerospace Sciences at Chanute 
Air Force Base, Illinois, was designated as the site of the 
service test. At that site, ATC was to provide space to 
house '25 PLATO IV terminals, students for both the instruc- 
tional materials in the conventional media and the parallel 
computer-based materials to be w»-itten, and a staff of eight 
full-tjlme members to write and implement the computer-based 
materials. The task of documenting these materials and 
their production was also given to ATC. Except for the 
PLATO IV terminals and their Communications costs, ATC sup- 
plied almost all of the physical and personnel resources 
for the test. 



11 



5 



Through a contract with the University of Illinois, 
ARPA furnished those resources which were not provided Ly 
ATC. ARPA's support consisted of the installation and main- 
tenance of 25 PLATO IV terminals along with the funding of 
communications to and time on the PLATO IV system's computer . 

» 

located at CERL. The terminals were to be equipped with 
slide selectors, touch panels, and audio devices so that a 
full capability for computer-based education would be 
available for the service test. ARPA axso' provided funds 



for CERL to train ATC and AFHRL personnel in the TUTOR 
language, curriculum development, operating- dyn:imics, and 
instructional capabilities of the PLATO IV system. The 
. Military Training Centers (MTC) group at CERL was to carry 
out the initial training for the Air Force service test and 
give advisory assistance in curriculum development and use 
of the PLATO IV system for the duration of the project. 

The remaining major participant in the service test was 
AFHRL. This laboratory is responsible for all Air Force 
research in the realm of behavioral sciences and personnel 
resources, Thus the task of providing overall guidance snd 



of designitig and carrying out an evaluation of the service 
J:e5t naturally fell to AFHRL. Specifically, according to 
• the research agreement, AFHRL was responsible for: 

1) providing direction and monitoring of the 
service test, 

2) deriving,, implementing, and monitoring mea- 
sures ofi student and instructor attitudes 
as well as performance, 

3) consulting on instructional systems develop- 
ment, instructional materials, and media 
devices,' 

^) consulting on the incorporation of PLATO IV 
into an on-going training environment and on 
the us% of PLATO and the TUTOR programming 
language , 
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^^^P^^ as the primary point of contact to 
ARTS on all matters pertaining to the Univer- 
sity of Illinois contract, 

6) providing quarterly letter reports of pro- 
gress, an annual siiinmary report of progress 

. and findings, Tand a final report at the con- 
..clusion of the experiment. 

Located at Lowry AFB, Colorado, AFHRL is about 900-" mMes^ 
from Rantoul, Illinois, the location of Chanute-AFB. Thus, 
AFHRL was tasked with direction of a project in which it had 
almost no^ investment i'ft material resources from a distance 
that made vigilant monitoring and accurate ..communications 
difficult at best. ^ 

The service test 'to, which these organizations were to 
contribute was orginally intended. to compare a convention- 
ally taught course:>with comput^V-based instructional mate- 
rials. The; General Purpose Vehicle Maintenance Course 
(GPVO- (3ABR47330) was selected for the purposes of this ' 
■comparison. 1 This course teaches a variety of skills 
including inspecting, servicing, testing, adjusting, • ' 
trouble-shoQting, and repairing general purpose automotive 
vehicles, such as light trucks and> automobiles . The variety 
of topics covered in t^he course, indicated that it would 
provide a good opportunity for the exploration of the 



'The verbal designation of this course from service 
test documents^ is unclear. Some refer to the course as the 
General Purpose Motor Vehicle Maintenance' Course; others, a 
the General Purpose Vehicle Re^pairman Course; and still 
others,' as the General Purpose Vehicle Maintenance Course. 
Despite the disagreement on nam"6s,-all <io'cuijients agree that 
the'' course is off icially designated by 3ABR47330. For sim- 
plicity of reference here the course will be designated 
GPVC for "General Purpose Vehicle Co.urse?" . 

13" 
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potentials and limitations of computer-based education. The 
course, as it was conventidlially taught, prfmariLy. employed . 
programmed texts and took 12 weeks for students to complete, 
the projected computer-based course was intended, to dupli- 
cate the objectives of the existing course point \for point 
while teaching them in a self-paced mode. By self-pacing 
the course, it was expected that substantial time savings 
would be realized. 

The eight staff members who were to develop the 
computer-based materials were to be under the authority of 
the GPVC supervisor. By having ■ charge of both the experi- 
mental and conventional courses, it was thought that he 
could insure that the computer-based materials taught the 
same objectives as the conventional materials. In addition 
to the guidance of the course supervisor, the Training- 
Research Applications Branch (TRAB), the School's evaluative 
branch, held the responsibility jointly with AFHRL of devis- 
ing an implementation and evaluation plan^for the service 
test. Thus, the PLATO staff was under the influence of 
three distinct sources-,-the GPVC chief, the TRAB, and AFHRL. 
F£ag mentation of the authority over the PLATO service test 
later contributed to several of the difficulties that 
plagued the project. 

The research agreement was approved in June, ^1972. ^The 
service test itself beg.an ort 15 July 1972 and was to con- 
tinue until 15 July 1975. The f^nal report for the test was' 
due 15 January 1976. '' • 

t 

Attempts to Establish a Service Xest Implementation Plan 

For the first 21 months of the service test, there was 
no finally approved implementatiorr plan . The absence of 
such a plan caused ^several management problems for the 
administrators of the service, test which will be discussed 
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in the section covering project manageme^nt before May, 1974. 
In discussing the establishment of the service test, a major 
concern will be to give a brief description and chronology 
of the several tentative implementation plans the project 
labored under. 

First plan. The first implementation plan was a simple 
elaboration of the research agreement. It was submitted by 
TRAB to AFHRL for review and comments on November 9, 1972. 
It called for the development of a PLATO based course paral- 
lel in subject matter to the already existing GPVC. The two 
courses would then be compared with respect to grades, reme- 
dial instruction times, course repetition and failure rates 
along with a variety of ad hoc measures of instructional and 
cost effectiveness. 

The major adrnin,istrat ive respon^rt^bility, both in daily 
matters and for execution of an infjlementation plan, resided 
with the GPVC supervisor. While the eight staff members who 
were selected to develop the PLATO materials were' directly 
under his* authority, TRAB also exercised some Influence' on 
the PLATO staff in an advisory capacity . Because the GPVC 

n5ijpe7^-soT-4--iTitreT-e^t~-in~-^ ser v ic e^te^t-was-no t ^- - 

great, TRAB took a more active part in the test than had 
been anticipated. 

An' active roLe for TRAB in the PLATO service t^st" ariose 
- oris ijnainy JY^ branch chief was the 

originator of the Request for Training Research* (RTR) which 
led to the Air Force's PLATO service test. His was the'plan 
that called for the computer-based versus conventionally 
based instruction comparison .(Whalen, 1972). Being in some 
sense responsible for the existence of the service .test, 
TRAB naturaliy^took an interest. This interest combined 
with its joint responsibility with AFHRL for an implementa- 
tion and evaluation plan for the test led to^the active 

15 



supervisory role^ that TRAB played in the CDE project. More- 
over, the fact that there was^a TRAB office on the base at 
Chanute allowed it to monitor progress of the project more 
clorfely than the more experienced supervisory personnel at- 
AFHRL. ^ • 

For four months from the submission of the first imple- 
mentation plan — from November, 1972 to January, 1973 TRAB 

personnel and AFHRL project monitors discussed the plan. 
They were unable to settle on a final version of the plan 
during this time. At the end of this period, the concept of 
implementing the service test as suggested by the RTR was 
questioned and, finally, rejected. 

The reasons for giving up the first. implementation plan 
involve problems with the start-up of the project, naivet* 
in drawing up the original RTR, and changes in the available 
student populations at Chanute. The fif-st problem relates 
primarily to management of the project and, hence, can be 
treated most appropriately in the section discussing the 
, serv'ice testes in'anageinent before May, 1974. It is enough to 
ilote here that the problems in getting^ the test underway 
were alone sufficient to make the first plan appear 
e xtre mely; d ifficult to complete at ' best^_ 

The outline of the servi.ee test set forth in the RTR 
was not thoroughly researched. While it was attractive at 
first to compare* directly conventional and computer-based 
instruction, it appeared later that such comparison was not 
desirable because of the n a t u r„e.^_:the_^_con v e n t i o n a 1 GPVC. 
This .course wa,s taught primarily through the use of pro- 
grammed texts. Hence, if the computer-baseS materials were 
to be developed so that they were comparable for^ evaluation 
purposes to the conventional materials, they would not give 
any demonstration of the capabilities of CBE beyond *the 
evident capability for simulating programmed texts. ' 
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Moreover, since the conventional GPVC was already effec- 
tively taught with programmed texts, a medium much less 
expensive than CBE, all hope of studying the cost effective 
ness of CBE would have to be given up. (Recall that deter- 
mination of cost effectiveness was one of the service test' 
original objectives.) Alternatively, if the PLATO-based 
materials were written with, a view towards fully, using the 
PLATO system's capabilities, the resulting materials would 
differ so much from the conventional that a comparison of 
the instructional effectiveness of the two media would be 
impossibl^e. This was the major theoretical difficulty with 
the original plan for. the service test. 

, As the Chanute service test got underway, other practi- 
cal obstacles to the successful imple'nentation of the origi- 
nal comparative evaluation concept appeared. The greatest' 
of these was a sudden d.ecrease in" the number of students 
taking the GPVC. In December, 1972, it appeared that there 
would not.>be enough students taking the course to provide 
data for significant comparison of tfte .conventional and 
computer-based courses. 

Finally, the service test be^an while the PLATO IV 
system was being built up to its present configuration. 
Thus, problems that tl^e managers of the service test had in 
organizing and managing their project were compounded with 
•frequent breaks in computer service of duration from a- few' 
minutes to several days and by delays in delivery of 
hardware. 

Thesp, problems, and others which will be detailed later 
in this section and in the following secti'on on early pro- 
ject management, convinced the administrators at Chanute 

that the original conception of the service test was infea- 

i( < 

sible. Simultaneously,, the monitors at AFHRL also began to 
question the original plan.^ According to the One Y.ear 
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Interim Report written by members of TRAB, these monitors 

felt that the two years that were to be spent developing 

.materials for the GPVC could be better spent in answering 

specific questions concerning computer-assisted instruction^ 

(Green, 1973, p.2)» ' / 

\: ' ' Second plan > On January 5, 1973, TRAB sent to AFHRL a 

memorandum outlining a new implementation plan for the ser- 

.vice test. The new plan would replace the tim^ that^ was t6 

be spent in developing materials for the GPVC with research 

into specific questions concerning the applications of CBE 

to technical training'.' According to the memorandum, this 

research wouJ.d emphasize "direct simplified , short-term 

research studies" (Main, 1973a, p.1).. As an indication of 

the scope of the research b6ing contemplated, the following 

research questions are'" excerpted from the memorandum (IbiT. , 

pp. 2-3): - 

Can PLATO serve as a part-:task training device 
in place of equipment? / " ^ 

Is PLATO'S ability to adapt to a variety of 
instructional, strategies an asset in Air Force 
^ technical training? . 

. Does the ability to employ drill and praxjtice, 
tutorial, page turning, or -copy frames: all within \ 
the same lesson enhance^ learning efficiency? \ 
Can PLATO be 'programmed to adapt' to the aptitude ^ ^ 
of the student going through the lesson? 
Is the microfiche .display an effective add-on 
to the capabilities of the CBE system? For what 

^ types of learning or performance behaviors are 

vmicrofiche slides useful or necessary? 

To. study these and similar questions, a plan for the utili- 
zation of the project ' s resources was offered. The plan was 
■"an^^^airtemplr to simultaneously an^sw^r some basic research 
questions about CBE while obtaining information about 
md'nagement^ and co~sf aspeci;*s-of using CBE as„an operational 
instructional medium. It thus represented an attempt to 
plan a service test which would satisfy th^ interests of 
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both AFHRI' and ATC- 

Thi'Si memorandum led to a meeting between Vepresenta- 
tives of the TRAB and AFHRL to set up guidelines for the 
implementation of new ideas for the service test* At this 
meeting the following points were agreed upon (Green, 1973, 
p. 3): ^ 

1) The TRAB would be responsible for the day- 
to-day operation, and HRL [AFHRL] would 
act as a second-level supervisory body and 
would publish the service test plan. 

2) There would be investigation >f the use 
of PLATO both fov computer-aided. instruc- 
tion and computer-managed instruction (CMI). 

3) Ap^proximately 80 hours of computer instruc- 
tion would be developed for CERL community 
college, application. ^ 

^) No attempt <would be made' to incorporate 
PLAT.O into the Motor Vehicle Maintenance 
Course thi^ough a formal -ISD effort. * 

5) Due to the inability- to create^^comparab le 
PLATO and non-PLATO courses,' no global 
comparisons, i.e., previously described 
dependent variables, measuring PLATO , 
versus conventional effects would be made. 

No attempts would be made to compare 
training times of lock-step versus self- 
paced training. 

6) Primary purpose would be to use PLATO .as * 

.a research vehicle di-ree-t-ed -toward use— oil- ______ 

CAI/CMI in an operational training setting. ~ 

When the original plans for the service test were ab.andoned, 
several policy questions which had been determined before 
were * rendered uncertain.^ These'points of agr;eement repre-:. 
sent an attempt to ^reestablish pjb'licy as well as';to, enunci- 
ate intended resear^ch questions. Also the first of these 
points gives TRAP the responsibility of publishing a service 
test plan . 

On the ba^sis of these agreements, AFHRL ""prepared a 
draft of an implementation plan and, in February, 1973, sent 
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it to Chanute for comment. The plan was intended as an out- 
line for an evaluation plan for the service test. Succeed- 
ing drafts of the plan were to make it applicable to the 
the day-to-day conduct of the test and its evaluation.' As 
with the previous agreement between AFHRL and TRAB person- 
nel, the plan was motivated by the concern of at once demon- 
strating the PLATO system in an operational setting and 
using the PLATO system as a vehicle for answering research 
questions regarding CBE methodologies (Pennell et al., 1973, 
p. 1). 

- The plan's reception "by TRAB was less than enthusias- 
tic. In .the view of TRAB, the plan suffered from several 

•deficiencies. Among these were the fact that its goals were 
not stated in measurable terms, that the olan did not ^ 
clearly set out the division of labor for the project, that 
it was esoteric, that it %^as too noncommittal and that it 
was general where it should have been specific and vice 
versa (Marn, 1973b). ' : , * ' 

Third plan . To settle points of disagreement and fur- 
ther refine the project's objectives, another meeting was 
held between TRAB and the AFHRL project morlitor. The even- 
tual outcome of this meeting was a draft of a new plan. 
Both TRAB personnel and the project monitor at AFHRL shared 
responsibility for writing this document. After some minor 
revisions, both parties approved it in May, 1973, nine 

^InointTTs ^irrr6^h-e--prt>;j^c-t-^r^d~i^ 
approval. This- final approval was never granted. ATC's 
reasons for withholding approval were based on the opinion 
that the plan's objectives were beyond' the capabilities of 
the service test. . Perhaps exhausted from the effort of 
writing implementation plans for the service test, TRAB used 
the basic ideas of this final attempt to guide the project 
until a major review was held in May, 1974. It was. 
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therefore, effectively the project's implementation plan for 
that period, of time despite the absence of full approval. 

This plan divided the project into three stages or 
phases. The first, which by May,' 1973, had for practical 
purposes been exhausted, was devoted .to the training of the 
PLATO authors. Experimentation with the PLATO system would 
be carried out during the second phase. During the final 
phase, instructional materials developed in the second phase 
would be used to Collect cost and management data. Five 
areas of interest were designated for investigation: 
instructional effectiveness of the PLATO system, instructor/ 
programmer characteristics, human factors and facility 
requirements, 'methods of management and conduct of computer- 
based training, and the economics of using computer-based 
instructional materials (Green jet al^. , 1973, p, 

Although this document was more substantial than the 
second plan, much of that weight was accouhted for by his- 
torical and contextual materials included in the plan. 
Many of the same criticisms whivch were directed towards the ' 
preceding plan could easMy be applied to this one. In the 
absence of any precisely defined objectives, it was directed 
at using the PLATO IV system as a research vehicle to gather, 
in as many areas as seemed accessible and with whatevt^r 
methods appeared feasi-ble. ' This attitude towards the ser- 
vice test is. expressed .clearly in the introductory discus- 
sion of the implementation plan: ■ 

< > 

The following sections have been provided to - 
give more detail on the specific techniques 
and methodologies to be employed in ac^om- 
^ pushing the service test goals. A caveat may 

be appropriate at this point, however. An ' - 
exploratory service tj-st of the kind planned 



, Mils effort must^o highly dynamic to pro- 
duce .ll>s most useful product. Therefore, 
methods and procedures detailed below must 
be considered mutab le - and subject- to changes 
dictated by aXheuristio' approach . (ibid., p.. 9) 

t 
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Thi_s statement reflects accurately the lack of confidence 
the plan's authors had in it. Better than the' pages spent 
in articulating the guidelines or, as experience showed, 
hopes for the service test, this statement characterizes 
the activit-y of the service test from its beginning until 
May, 197^*. 

As stated before, A7C never "gave its final approval to 
this plan. In the absence of ar.y other plan, however, the 
managers of the service test at Chanute AFB used it as a 
basis for administering the project until May, 1974. At 
that 'time the project was given a full review which resulted 
in a drastic change in direction. Because this change was 
so radical and' because it resulted in the most productive 
period of the service test, the change and its consequences" 
are discussed in a separate section on the ISD management. 
Here, it is sufficient to note that after -^the May ^9^H 
review, the service test's goals in terms of products and 
timelines, were- precisely, defined. -MQi-eover, the major, 
effort was to be devoted tp producing, operational instruc- 
tional materials on the PLATa system rather than using PLATO 
lessons individually as research vehicles. Research under 
the new plan was. to be confined to evalua^tton of the effec- 
tiveness of a computer-based medium in an operational 
setting. • ' ' • 

Inherent Defects In The Service Test ' . • 

Before turning to other aspects of tne establishment of 
Chanute's CBE Service Test, i^t Ts necessary to gddress some 
comments' to the situation whiqh effectively impeded the suc- 
cessful writing of an implementation plan for almost the 
first half of the project's duration. Interviews with ser- 
vice test evaluators, managers, and authors along with 
observation of the test throughout its duration ^ave' 
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indicated several contributing factors to this circumstance. 
Those which were inherent in the service test, and 'hence 
directly affected the establishment, will be dealt with 
here. Other factors will be treated as they become relevant 
to the succeeding discussion. 

The fundamental source for indecisi veness with respect 

'to the service test's objectives can be found in the differ- 
ent missions of the two Air Force agencies responsible for 
the test^ AFHRL .and ATC . On the one hand, ATC ijs concerned 
with '■'le technical training of servicemen. Its interest, 
therefore, was ,to discover how the PLATO system would 

. perform as a technical training medium. AFHRL, on the 
other hand,, with- it.s respart^ibility for all "Air Force 
behavioral science>esearch knd its new commitment to the 
AdvancBd^ Instructional System (AIS), a large CBE system 
under development -at Lowry^AFB, was naturally concerned to " ' 
. discover as much as possible about computer-based media. ^ 
, . Thus, ATC*\^' interests in the PLATO system were based pri- 

^inariLy in its capability to deliver actual instruction and 
secondarily with' the potential of the 'system to support 
basic CBE research; priorities of interests of AFHRL in 
• these two "areas were reversed from those of ATC. 

Practically, the interest of AFHRL in relatively pure 
uBE rerjarch manifested itself in droppin-g the first imple- 
mentation plan. If implementing the service test as 

^-phe contract far. the AIS system was awarded in May, 
1973. AIS was not only to off^r self-paced, individualized 
computer-based instruction to Air Force trainees but also 
to manage their training by selecting materials and media 
allocating resources- thousands of students. The plans for 
AIS called for a system which possessed strong resemblances 
to the PLATO IV system* 
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originally planned in the GPVC was impossible, surely anoth- 
er course could have been selected for a similar sort of 
service "-test. According to personnel invol-ved with the ser- 
vice 'test af that t.ime, no alternative course was sought. 
ATC»s influence and prejudices are evident by the fact that 
it withheld approval of any of the implementation plans 
which were drawn uj)- by AFHRL. 

The practical effect of this indecisiveness for the 
day-to-day management of the project cannot be overesti- 
mated. Without an implemention Rl^n, TRAB personnel who 
were administering the test at Chanute had no real guide- 
lines with respect to both products l.kJ deadlines. Inter- 
views with members of Chanute PLATO project indicate that 
the absence of clearly enunciated, fundamental objectives 
and the' consequent lack of direction from those ^agencies^ 
direqtly responsible for the service test severely under- 
mined the basis of authority of the project's^ local 
managers. 

In addition to the clash of fun-damental interests of 
AFHRL and ATC in the PLATO service test, neither of these 
organizations had had much experience with computer-based 
education. Moreover, since Chanute was the first military^ 
training center to use ^he PLATO system',' CERL could provide 
little in the way of -guidel ines or advice in the preparation 
times for technical training materials. The* practicalc 

effect of this lack of experience was* that none of the ^ J 

people attempting to write an implementation plan had a 
clear concept of what was feasible in a three year PLATO 
service test. For example, being novices with the system, 
they had no reasonable estimate of how many manhoii'rs per 
hour of student instruction would be needed to produce even 
the simplest sorts of lessons. Thus, any estimate of the 
magnitude of the project^s total output was purely guess- 
work. ' " • 
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As an example of this sort of guesswork, consider that 
when the- project began it was proposed to use the PLATO 
system to duplicate most of the non-laboratory, instruction 
in the GPVC. Thus, it was hoped that about 173 hours'of 
instruction would be producedv As far as is known no consi- 
deratioa was given at that time to the work load that such 
an objective would place -on the\ PLATO authors. In any case, 
less than sioc months later the objective for the service 
test was reduced to. 80 hours of "CBE lesson material. This 
, objective was essentially halved again in May, 1973, when 
TRA© and AFHRL made their final attempt to wrile a joxnt 
implementation plan. This last estimate is unique among its 
predecessors in that it is based on an estimated rate of . 
lesson^ production, namely, that observed in the early por- 
tions of the service test. If objectives about particulars 
such as the quantity of lesson materials 'to be produced were 
so casually madej it is easy to see how the other objectives 
for which there was no^ ground for making production esti- 
mates, would probably be more unrealistic. 

Philosophical interlude . In view of this naivetft with 
respect to CBE, the TRAB managers of the service test should 
not have been given the license by the AFHRL project moni- 
tors and evaluators of being ^able to change the project's 
objectives in the face of day-to-day cont ingehcies . Such a 
"policy would be most likely to lead to so many changes' in 
direction that at its conclusion the project would havejpa^e 
very little progress in any given direction. Given tt^is 
inexperience, the -project planners would. have been ^l^st 
.advised to follow the recommendation of Ren* Desoartes on 
how to embark on projeo^ in which -the way tp'proceed is 
less than clear. In sucnc^^-s-y-Jifi^ar'tes says that one 
must early establish some rule for action and resolutely 
follow it. Even ifgjhe rule is uncertain and if it is known 
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to be doul/tful, 
one * would^ be 

following the example of travelers, who, rinding 
themselves lost in a forest, know that they ought 
not to wander first to one side and then to the 
,c/tiier, nor, still less, to stop in one place, but 
ytindorstand that they should continue t;o walk .as 
/straight as they can in one^ direction, not diverg- 
/ ing for any slight reaspu^^even though it was 
/ possibly chance alone that first determined them in 
/ their choice. By this means if they do not go 
/ exactly where the'y wish, they will at last arrive 
/ somewhere in the end, where probably they will be 
better off than in the middle of a forest. And 
thus since often enough in the actions of life no 
delay is permissible, it is very certain that, when 
it is beyond our ^power to discern the opinions 
which carry the most truth, we should fpllow the • 
most probable; and even although we notice no 
greater probability in one opinion than in the 
other, we at least should make up our minds to 
follow a particular one and afterwards consider it 
as no longer doubtful in its relationship ,to 
^p,ractice, but as vei>y true and very certain, 
inasmuch as the reason which caused us to deter^ 
mine upon it is known to be so. (Descartes, p. 96) 

The sagacity of Descartes' advice was not called into 
doubt by the Chanute service test, which changed implemen- 
tation plans at least three times during the first half of 
its intended duration and never finally fixed on any one 
plan,. Before a final management plan was adopted, the major 
accomplishment of the service test consisted in the training 
of the authors. Although the lessons they developed did 
serve as a basis during the second .phase of the project, 
these lessons had neither the cohesiveness nor polish to be 
implemented as bona fide CBE lessons. The effect of 
changing plans so often was that any progress made during 
one period was canceled out by progress in another. 
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it must be treated as certain. In this way. 





S election and Training of PLATO Authors 

The selection, and training of the PLATO authors was not 
delayed or subs'tanti&lly impaired by the absence of an 
implementation plan. By early August, 1972,. four civilian 
and, four military authors had been identified by the mana- 
gers of the service test and were ready to begin' training. 
These eight authors were' originally put under vthe supervi- - 
sion of the chief of the GPVC along with 31 other personnel 
authorised as instructors in the course. ' The fact that the 
authors were* under the management of the GPVC chief and of 
TRA3 personnel simultaneously was the source -of much diffi- 
culty in the early administration of the service , test . 
X The. branch, chief of TRAB" who wrote the original RTR was 
responsible also fdr selecting authors for the PLATO service 
test. Youth was one of * the primary qualities he sought in 
potential PLATO authors. He felt, that relatively young 
instructors ^would be sufficiently aclaptable tp revise their 
ecjllcational outlooks and work habits to suit the needs of 
using the PLATO system as a teaching medium. He- hypothe- 
sized that not only would the authors have to use n'ew 
Instructional technitjues but that they would also have to 
master ia .computer. programmin.g language and the protocols of^ 
working wijth a computer system. . ' / 

The TRAB. branch chief eraployed t,wo 'other crdtet^ia in 
selecting the project'3 authors* These two criteria were 
amoutft of education and willingness to stay with the PLATO ' 
service test throughout its duratio'n. Selectiop o'f the most 
highly educated personnel available, in .the opinion of the 
branch chief, would increase the likelihood that the authors 
would possess the intellectual tools .and discipline to mas- 
ter the complex skills needed ,to author materials on the 
PLATO system. Learning these' skills^^' moreover, involved so 
much training that the loss of an author during the service* 
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test would' be very costly to the project' both in terms of 
the loss of the investment in the author but also in terjns 
of the time needed to train" a replacement. 

The criterion that the authors be sul?jecJt' matter 
experts 'in the intended areas of CBE lesson development was 
not of high priority in author selecti^. Since the pool of 
instructors from which the authors were selected consisted 
of GPVC instructors, it, was assumed that any of the- candi- 
, dates would be highly knowledgeable in the subject-matter of 
that course. The fact that a premium was placed on youth, 
however, resulted in the selection of some authors who did 
not, have much experience with teaching or teaching the 
course's subject matter. • ' 

f It is not known whether experience in the areas of 
instructional design and computer science was considered an 
^important criterion ih authdr selectidn, but, .because hf 
. available candidates,- could 'not be satisfied^ or whether 
these characteristics were not considered important. In- any 
case. Table 1 shows that the authors selected " collectively / 
possessed no experience^ in tne first of tte-se-^ks'and only 
a little in the second. Finally-, t'he TRAB branch chief was 
constrained ^o select four civilian and four military per- 
sonnel for the PLATO author staff. Table 1 summarizes the 
characteristics of the staff. 

Table 1 shows that some of the primary selection cri- 
teria were well' met by the authoring staff. With respect' 
to getting s'taff members to stay for the duration of the 
project, hovever, there was much less success. Two of the ' 
military authors left the Air Force, and hence the project, 
after the project was little more than a year old. Another 
military author leTt the project after 18 months. Also one 
civilian author left th^ project after two years. Because 
of a scarcity of qualified personnel and bureaucratic 



TablQ 1 

Characterist?cs of the Chanute PLATO Authors in July,, 19.7.2- 



Author Age Education 

' 1 3^ H.S. graduate 

.2 32. M.A.> (Political Science) 

M.A. (Sociology) + 9 hrs. 

3 29 H.S. graduate + 

2 years college 

4 ^ 26 B.A. (Industrial Ed) 
'5 25 B.S. (Biology) 

6 25 H.S. giaduate + 

1.5 years college , 

7 23 B.A. (Industrial Ed) 

8 23 B.A. (Physics) 



, Years 
teaching, . 
experience ' 

'•• 5 

■ 3 

9 

3 
1 
1 

2 
1 



E/xperience 
with , 
computers 

keypunch operator 

none 

none 

none ^ - 
keypunch operator 
none 

none 

1 year of college 
computer science 



Rank 
Master Sergeant 
GS-9 

GS~9 
Staff Sergeant 

GSr9 

Sergeant • 

Sergeant 
GS-9 



a 

. None of the authors had any experience in 'writing programmed 
instruction of any sort. 

Pennel et al., 1973, p. 6a. 
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•difficulties, satisfactory replacements were generally not 
raa'de for those authors who departed from the project before 
the intended completion date of July, 1975. *' 

Acquisition of PLATO IV terminals and Peripheral Devices 

The Chanute PLATO service test was allocated a total of 
25 PLATO -terminals. The delivery of these terminals was to 
be completed by the end of the second year of -the service 
test. Each terminal was to' be equipped with a random dccess 
slide selector (Francis, 1976, p. 4), a random access audio 
playback device ( Ibid ., 1976, p. 38), and a touch p^nel " 
( I^id .,.1976. p. 25) . . . " - ' 

, Because of manufacturers' delays, CERL repeatedly 
revised times the schedules for the delivery of terminals^ 
and peripherals. Th"^ inability t^o follow any of these 
schedules stems primarily from delays in delivery of hard- 
ware to CERL from various manufacturers. The delayjs of. the 
•manufacturers on the other hand may be attributed to the 
fact that much of the hardware ' for the new PLATO IV system 
was being manufactured for the first time so that' these 
manufacturers were encountering the problems that usually 
accompany the assemb.ly of new' products . For Chanute, the 
source of these delays was not important. All thSt con- 
cerned this site was that the delays did exist, that they 
were long, and that they impeded the ^ability of the newly 
trained PLATO authors to develop their skills in using the 
PLATO system. • ' ' . 

Chanute's first two terminals arrived two weeks late in 
September, 1972. This delay- forced, the authors to' travel 
daily to the University of Illinois for two weeks to con- 
tinue to use the PLATO IV terminals there (see Appendix 1).' 
The authors had just completed their TUTOR training at that 
time. Consequently the importance of getting extensive 
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exposure to the PLATO system 'far outweighed the inconve- 
nience of working in an unfamiliar and temporary environment 
and of comniuting between Chanute AFB and CER'L, a trip of- 18 
miles each way. 

Another ill effect of the slow delivery of PLATO ter- 
minals was that the eight Chanute authors were split into 
three groups. with each group working one of the three work- 
day shifts. (This was possible since PLATO system service , 
wa? available 22 hours a day.O As the shifts were originally 
established and remained .through June, 1973, two shifts each 
with three authors did not interact at .all. Not only did 
this arrangement hinder communication within the authoring 
group, it also ofterr> rendered ineffectual the attempts of 
the TRAB personnel to monitor' and guide the progress of the 
authors. In June, 1973, the shifts Were' dif f erentl y stag- 
gered to avoid some of these problems.. Jt was not until 
January, 197^, when the eighth terminal was installed that > 
all of the authors could work the same shift. 

The slow delivery of terminals was perceived by the on- 
site managers as one of the project's biggest problems ^ - 
during its initial- stages : 

In the beginning of a project, on.e a*dditional ter- 
minal can make all the difference as far 'as avail- 
able terminal hours an^i flexibility of scheduling. 
Never knowing when the next terminal would be 
delivered and always being promised that it would 
^ be there the next week added more .to the list of 
- frustrations. (Green, 1973^ p. 11) 

The inability of CERL to deliver Chanute's terminals 

y 

according to most of 'the proposed schedules also damaged the 
confidence of the project adminstrators in CERL's promises 
for delivery of other hardware. In the case of the touch 
panels, the lack of confidence had a definite impact tfn 
courseware development. ^ Although enough panels were 
installed for author use, the project administrators' refused 
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'to allow their authors to in-corporate , use of the touch panel 
into the PLATO lessons even though delivery of a sufficient 
number was promised by CERL before the students would be 
using them. Only with all of the touch panels installed 
would such reliance on them in the PLATO lessons.be per- 
mitted. For this reason, touch panels were not used in any 
of Chanute's operational lessons although each terminal was 
eventually accoutered with one. 

The situation with audio devices was much the same as 
with touch panels. Since the site was not completely supr 
plied with these peripherals, it was decided not to use them 
in^the CBE lessQns. In fact, for the entire duration of the 
service test and the subsequent operational ijse of phe PLATO 
lessons, there was never more than one audio ^device at Cha- 
nute, and it was seldom, if ever, used. 

With respect ^to the slide selectors the *case was some- 
what more fortunate. A 'full complement of these was 
installed early enough so that the decision to use micro- 
fiche in the CBE lessons could b^ made. The slide selec- . 
tors ,. nevertheless , required ^ frequent' on-site maintenance 
and were not found to be reliable by the Chanute authors 
(Francis, 1976, p.* 30). An extensive "descript ion of Cha- 
nute^s experience in preparing microfiche for their lessO;ns 
may be found on pages 19-32 of Francis (1976). 

Service Test Managers 

. The multiplicity of interests in the Chanute PLATO IV 
service test -has already been detailed in the discussion of 
the research agreement. It should be recalled that the 
PLATO authors were -of f icially designated as instructors in 
the GPVC. At the same time, the entire PLATO service test 
was unde*" the general guidance of AFHRL which was trying to 
exercise its supervisory duties from a distance of 900 miles 
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from the test site. Finally, TRAB personnel located at Cha- 
nute AFB acted as on-site project managers sometimes ^expe- 
diting directives from AFHRL,' sometimes collaborating with 
AFHRL in formulating project plans, and sometimes taking the 
initiative in administering and directing the service test. 
All in all, it appears that there was an attempt to. compen- 
sate for the absence of^ an implementation plan with a sur- 
feit of project administrators. 

Of ,the three organizations directly involved wj.th the 
service test, TRAB was by far the most important. The ori- 
ginal Research Agreement designated TRAB as the 

- Technical School OPR [Office of Primary 
Responsibility] for coordination of the - 
experimental service test with ATC/XPT, AFHRL, 
ARPA, and the University of Illinois. This , 
office will monitor the research program by 
developing a programmed action directive for the 
duration of the three-year research agreement. 
(Training Research Agreement, I972) 

What monitoring a research program by means of a . "programmed 
action directive" may mean is not altogether^ clear. How- 
ever, the TRAB Branch Chief and. his staff members were 
de facto project officers of the service test'from its 
inception until May 

By "project officers" something less than might ordi-^ 
narily be understood is meant, 'for in any effort to direct 
the project, . TBAB__p„er.sonneL-had-to-^contend with the -fact 
that the PLATO authors were under' the supervision of the 
GPVC branch chief. Without the cooperation of the GPVC 
chief, therefore, the influence of the TRAB persor^nel was 
limited* Again, without some convergence of goals- of ATC 
andi AFHRL, the administration of the project on-site was 
indecisive at best* An awareness by the TRAB personnel of 
the diversity of influences affecting TRAB is apparent in 
the following quote from a documentation of early^ project" 
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problems: 

It seems apparent that TTOE^ personnel -are the 
critical link among all contacts concerned with 
the PLATO effort. Everyone involved- with- the 
prcsram at whatever" level seems to have their 
own interests uppermost, in mind when any problem 
about PLATO is brought to their attention. 
(Green, 1972, p. 1) . 

Clearly, since- its role in the service test was subor- 
dinate to that of AFHRL and ,AfC, TRAB could not administer 
the project" effectively witl^out some convergence of inter- 
ests of those two agencies and without an approved implemen- 
tation plan. There were other conditions that together made 
effective leadership ' of the project, beyond the pr^actical 
capabilities of TRAB. Amqng these- conditions wer-e frequent- 
personnel changes, in TRAB and, consequently, Tn the PLATO 
project's administrator during the first two. years of the 
test. These changes along with an assessment of their 
impact on the projept is contained in the section discussing 
management in the first fwo years of the project. ' 

Since the le'&dership of the project changed so often in 
its first two years it would be difficult to categorize the 
(Qualifications of the TRAB personnel. In interviews with 
members of TRAB's staff, however, one deficiency in particu-- 
lar was emphasized as -being especially detrimental to the 
success of the test. During its first two years, TRAB nei- 
ther had any staff. member s experienced in curriculum design 
nor ready access to Air For^e personnel with experience in ', 
this field: This- deficiency was. characterized by a- member 
of the TRAB staff who had held a position of high responsi- 
bility in the administration of the project as being the 
single most • important factor in contributing to the relative 
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lack of progress in the project»s first two years. He made 
this assessment in comparing this problem with those of 
trying to develop an implementation plan that would satisfy 
both AFHRL and ATC and of trying to undertake an ambitious * 
curriculum development project., on a newly assembled innova- 
tive CBE system. • * * 

It should be, recalled, that none .of the pLaTO authors 
had a significant ^background* in instructional design and 
that none of 'them had any background in developing lessons 
for CBE. It is then easy to see that with this naivet* with 
respect to CBE instructional design on the part of the 
authoring' staff , the absence of experience in curriculum 
design could be critical to the project»s outcome* It is 
merely speculation to attribute som^* of the problems of the 
project to the absence of a curriculum designer At another 
ARPA/PLATO site which was administered^ by an individual with 
ten years of * curriculum des^ign experience, fewer of these 
problems arose (Himwich, 1977). This relative competence 
was not only apparent in daily management of. project affairs 
but also in allocating responsibilities to the staff *s 
members so that the project^s long range goals could be 
achieved. 

Summary 

The Air Forceps PLATO service test had several diffi- 
culties which were either inherent in the structure of the 
test or could be -attributable causes beyond the control 
of the project»s planners and administrators. Since these 
problems were the causes of ,severalv^def iciencies in the * 
establishment of the test and because' they continued to 
plague the project throughout its duration, it is fitting by 
way of summary of the project establishmeat to enumerate 
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'them here.. 

The problems that arose in the service test seem to be 
from three fundamental sources: 

1) The two Air Force agencies — ATC and APHRL— 

^- which had primary responsibilities' for the ser- 
vice test had different aspirations for the 
2y«mn"'® °^ project. ATC was interested in ' 

u?; ^ operational teaching devi e 

-while AFHRL' tended to view the PLATO system 
as a vehicle oi' basic CBE research. 

2) The service test, began at--the same time that 
the new PLATO JV system was being assembled. 

3) Neither among the supervisors of the project^ 
nor among the PL'ATO authors was- there,v any signi- 
ficant experience in curriculum and instruc- 
tional design. > , • 

The first of these has been discussed as, a primary 
reason that no implementation plan was officially adopted 
for the project.' Also, as will be more apparent in sue- • 
ceeding sections of this chapter, it made Effective admini- 
stration of the project at Chanufe AFB.by TRAB difficult. 
However, it is clear that the task of managing the project 
locally and 'simultaneously serving the interests of ATC and 
AFHRL could only be successfully completed with the aid of 

xve tslent or of exceptionally good 
fortune. Chanute .was blessed with'' neither . ' 

One of the manifestations of the second of these pro- 
blems appeared in the delays in the delivery of terminals 
and peripheral hardware. These delays, in turn, were an 
'impediment. to the project authors in practicing skills they 
had just been taught in author training at CERL;- Finally, 
with the start-up of a computer system as complex and large 
as the PLATO -IV system some difficulties with regard to 
establishing reliability must be expected. Being the first 
ARPA/PLATO site and , one of the first large site^ remote fr.om 
the University of Illinois, the Chanute service test fell 
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victim to a number of problems that would not and did not , 
trouble later sites. These problems will be discussed 
wherever appropriate in succeeding sections of this chapter 

The absence of experience in cui^riculum and instruc- 
tional design in the project had its main impact in the 
manner in which the project authors were adminisj^ered . Tt 
is, therefore, sufficient to emphasize this fact here and 
discuss it in- the section on project management'. 



SERVICE TEST MANGEMENT BEFORE MAY, ^97M 

The management of Chanute's PLATO service' test-> under- 
went several changes in the. first portion of the project. 
These- changes are in part attributable to turnovers in the 
Staffing of TRAB -which had the responsibility of locally 
monitoring the project and of ^coordinating the efforts of* 
the- agencies • that were involved with the pVoject. Other 
changes occurred as some workable scheme for administrating 
th«. daily progress of the authoring staff was sought* These 
changes are important bec-ause they revealed defects in .ihe 
original structure of the service test and because they had 
an effect on the authoring staff* The first part of this 
section discusses <the^ evolutibn of the'- service test manage- 
ment. The second part^will npte other . influences on the 
management of the project. Finally, the lesson development 
processes during this phase of 'the service test will be 
described. 

Evolution of the Project Management before May, 1974 

Vmile the ^leader shj p of TRAB could not be well directed 
without a convergence of interests of ATC and AFHRL and 
without an approved implementation plan, t\\e fact that TRAb 
underwent a number of personnel changes during the early 
stages of the project also contributed to TRAB^s indecisive 
leadership at the site level. Which> if any., of these 
changes were mo§t detrimental is a matter for speculation. 
However, it can not be doubted that, taken separately or 
together, they did have adverse influence on th.e project^s 
outcome. 

The first change in local leadership of the project - 
occurred shortly after the authors had completed their PtATO 
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training at CERL^in September, 1972, At that time, ,the TRAB 
branch chief left Chahute AFB, This individual, by virtue 
of his proximity to the University of Illinois^ had been 
aware of the development of the PLATO IV system at "CERL; 
Being excited by the potential of a powerful computer-based 
educational system, he initiated the project's RTR% Also, 
the central idea of evaluating the • effectiveness of the 
PLATO IV system by duj)licating the already existing GPVC was 
his. It is not remarkable that entiiusiasm for this approach 
to the service test diminished after his departure in the 
face of other difficulties to that concept's implementation. 

''At 'the time of the departure of the first project offi- 
cer, one of the psychologists from TRAB's military personnel 
was assigned as project officer working one-half time on the 
PLATO project. TRAB's office and that of the. PLATO authors 
were ^ not of sufficient proximity to each another for the new 
project officer to get a satisfactory feeling for the needs 
of the project. To remedy th^s situation, he obtained a 
desk within the PLATO group. Once he began to have r^re 
contact with the PLATO authors, however, his involvement 
with the project increased. By January, 1973, he was work- 
ing full-time with the proje'ct (Green, 1973, p. 6). 

Increased commitment to the PLATO project of all mem- 
bers of TRAB followed that of the project officer. TRAB's 
staff consisted of two military psychologists, a civilian 
training specialist, and a civilian secretary . One of the 
psychologists was already acting as project officer. By 
early April, the other psychologist and the training spe- 
cialist were spending a significant portion of their time 
with th** PLATO project also. At that time, in fact, serrous 
consideration was given to the idea of moving the entire 
TRAB^^ffice to a room adjoining the PLATO section (Green, 
1973, p/8). - ^ ' ^ 
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Vfliile TRAB's inyolvenjent was increasing, some diffiaiii- 
tiefi between the^ PLATO group and the other instrjjctarf^orn^ 
the GPVC arose. From the out'set, the PLATO^st'aff,^ though 
part of the GPVC staff, was vie^ed^witfTscime envy by their 
colleagues who taught the^an-ventjiohaT^ourse . It was felt 
that' since the ^^PL^XO- -staff did not have to instruct as con- 
ventional lecturers tha^ .their situation was preferable to 
tha^i of'the other instructors. Moreover, early in the prp- 
ject there -were onl^y^wo terminals installed at ^hanute.- 
Thus', only two authors could tise the PLATO system at a time 
while the other six were left to ^lan their lesson's at their 
desks. This behavior was judged to.*be non-productive by the 
conventional instructors and the (JPVC supervisory personxiel. 

Early in 1973, because of hardware and software acqui- 
sition and modification, the PLATO system was unavailable * 
for six weeks. Restricted entirely to desk work, the labors 
of the CBE authors 'seemed to be even more unproductive than 

th.ey had appearecj earlier. This appearance, it'was argued, 

' ' • /•. 

had a demoralizing effec't on the conventional instructors. 

^ t* * 

.To remedy the apparent inequity, the PLATO authors, military 
and civilian alike, were ordered to take part in custodial 
duties that required several hours a day of stripping, 
cleaning, and waxing floors*. These duties were even main- 
tained after the PLATO system was once, again available on a 
reliable ' basis. In fact, it was only *after hard and long 
argumentation from the TRAB project officer that the GPVC' 
Branch Chief was' cpnvinced that the PLATO staff had enough . 
•to do without these . additional duties . 

- The friction l^etween the PLATO authors and the other 
GPVC instructors wSs heightened by the ambiguity in the 
leadership of the project. On the one hand, the GPVC chief 
was officially the supervisor > of the CBE authors, while on 
the other TRAB personnel bad taken the' active, role in 

'41 • 



f ' 

administering the ..project. Again, the GPVC chief ''knew 
little, if anything, about the project" (Green, 1973, p. 8) 
while raembers of TRAB had taken the time and trouble to 
learn TUTOR, the programming language used on the PLATO 
sys:tem",- as well as other efforts to help the Authors in 
undertakirig the task of developing CBE materials. Thus 
TRAB, with no official sanction to supervise the CBE 
authors, was in fact doing so while the officially desig- 
nated supervisors, were taking -very little interest in any 
aspect of the service test. 

In early 1973, an attempt was made to alleviate the 
the ambiguity in leadership of the PLATO projects and to,, 
reduce somewhat the friction that had developed between the 
PLATO authors, as GPVC instructors, and the other course 
ins;tructors teaching the conventional GPVC. To-,solve these 
f)roblems, discussions were begun 'concerning the feasibility 
of- transferring the authors to TRAB. This transfer was in 
fact made with the understanding that the PLATO authors 
would revert to the supervision of the GPVC chief once their 
materials were completed and were ^being used operationally. 
With this change in the management of the project and the 
involvement of the entire TRAB, the authors were under a 
singleSlocal leadership. 

-Vl^en TgAB began administering the daily affairs of the 
service test, the- need for a larger research component for ' 
thp project .became apparent to TRAB personnel (Green et al . , 
1973, p. ^0). To assist in monitoring this need, TRAB urged 
AFHRL to meet the terms of the original agreement between 
ATC and AFHRL and provide an on-site AFHRL representative 
(Green, 1973, p. 9). In June 1973, AFHRL hired an indivi- 
dual to serve in this role. The view of TRAB towards this 
appointment was quite sanguine. TRAB anticipated that "This 
action-)will insure that research needs under jurisdiction of 
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Air Force Systems 'Command and operational requirements of 
Air Training Command^ are properly balanced*" (Green et al . , 
1973, p. 40)^. Ap immediate benefit of the appointment was 
that communications between the service' test site and AFHRL 
at Lowry AFB, Colorado, were improved. 

For service test monitor and liaison, AFHRL selected 
/i#fi individual who had been an instructor at Chanute AFB for 
five years. As an instructor he had probably gained some 
experience of with instructional design and implementation 
of instruction. However, there is little in this background 
that would indicate a competence in evaluation, management, 
or curriculum design. With ' experience only in the area of 
instruction, this individual was not prepared to insure that 
the service test maintained a "proper^ balance" between 
research and operational concerns . 

The management role of the AFHRL monitor was very 
important during the first phase of the service test. 
Despite his inexperience in the important areas of educa- 
tional research, curriculum and instructional design, and 
management, he and the TRAB project officer were very nearly 
peers in these areas. Thus, his opinions, which were 
already invested with the weight of his position as AFHRL 's 
repnesentiv_e, generally influen.ced^^the project officer's 
decisions strongly. Together the AFHRL representative and 
TRAB project officer administered the servide test from the 
time the former assumed his duties in June, 1973,^until the* 
first phase of the project was formally reviewed in May, 

^This statement is of significance not only as documen- 
tation of what role the AFHRL liaison was expected to 
fill but as an indication that TRAB w.-^s aware of the 
difficulties in trying to administer a project that was 
to be simultaneously research and operationally oriented. 
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1974. For this reason, the administration of the project 
during this period may be justifiably attribute^ to the 
"management/evaluation" staff. 

During the ,tenure of this management/evaluation staff, 
the authors ' essentially operated autonomously. For most of 
this time, the project only had four PLATO -terminals . To 
maximize the time that each author could work on the, PLATO 
system, the staff worked three daily shifts. For the 
authors working under 6uch conditions decreased somewhat 
€roup interactions that are useful in solving problen^.s that 
arise in l^esson development and in utilizing the variety of 
talents in an authoring group. For the management/evalua-^, 
tion staff, splitting the authors into three shifts meant 
t|iat the authors often worked withdut supervision and that 
their progress was difficult to monitor. 

Having established the habit of working autonomously 
the authors naturally resisted attempts to establish any 
regimentation in their work habits. This resistance 
amounted to more than the' mere distaste for supervision. On 
one occasion, when the project was well into its second 
year, the AFHRL monitor proposed an implementation plan. A 
— st-rong negative reaction to this plan by the service test 
authors was sufficient to prevent him from formally propos- 
ing the plan to AFHRL and ATC . At other times, the authors, 
both individually and in groups, undertook to reform the 
project themselves. With respect to providing some direc- 
tion to their work, their efforts were as ineffectual as 
those of the management/evaluation staff. It is,, however, 
noteworthy that they undertook such initiatives while 
occupying subordinate positions. It is a token of the lack 
of direction and order that pervaded the project during this 
phase. 
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With the possible exception of the first writer, of the 
RTR, the succession of TRAB project officers and the on-site 
A.FHRL representative were all actively engaged in graduate 
degree programs at the University oT Illinois. Two of the 
TRAB project officers were involved in doctoral programs. 
One of these officers felt so acutely the demands of simul- 
taneously managing a CBE curriculum development project and 
of pursuihg an advanced degree that he left the Air Force to 
become, a full-time graduate student. Anothfer took leave of 
his Air Force duties to devote his energies full-time to his 
graduate program. 

It is impossible to assess the overall impact that 
enrollment in graduate degree programs had on the quality of 
the management of.^the Chanute' service test. Enhanced back- 
grounds in education and evaluation were valuable to the 
management of the project. However, the courses and studies 
were directed at long-term career objectives and so were 
more extensive than mere supplementation of knowledge needed 
to direct the service test. Also, in two cases the commit- 
ments of a project officer ^ere radically altered for the 
sake of continuing graduate studies. It is, therefore, 
plausible to conclude that their studies on the whole 
detracted from' the performance of their duties as project 
managers. How much these' duties were affected can only be a 
matter of speculation. 

From interviews with service test authors, it has been 
learned that while members of the management/evaluation 
staff were enrolled in graduate degree programs, they 
actively discouraged the CBE authors from undertaking addi- 
tional coursework themselves. Even though in some cases 
the authors were interested in taking courses which would 
have increased their value^ to the service test, at least one 
of the PLATO authors had his desire to take courses 
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-categorized as self-serving. , (The_ completion of additional 
c'oursework facilitated promotion.)' Such criticism appeared 
to the authors to be hypocritical in view of the^ fact that/ 
some^members of the management/evaluation staff were ' 
pursuing degrees which would enhance their professional 
standing, c Despite these discouraging conditions, after 
appealing to GPVC supervisors, at 'least two of thB original 
eight authors took coursework while working for the service' 
test* 

The question of whether or not the authors' perceptions 
were accurate is not relevant to this report. What is 
important is how they perceived the managers of the service 
Jest\ ^That authors' perceptions were as described is a 
token of the poor esprit de corps of the service test per- 
sonnel. To outside observers of the project, the morale 
usually appeared bad. Such an assessment seems to be. corro- 
borated by the seryice test manager/evaluators who once, in' 
conversatio'n, characterized the attitude of the authors as 
one of "open rebellion" to the administration of the 
project. 

P 

Lesson Development Procedures 

Those interested in planning a CBE curriculum project 
would never consciously imitate the lesson development pro- 
cedures employed at Chanute before May, 1974. ^5 These 

^Planners for^he PLATO projects at other military 
training centers frequently included an inquiry into the 
history of the Chanute service test as an essential part of 
their preparations. That this project was beset with many 
problems was well known. They hoped that a thorough know- 
ledge of these troubles would help them to avoid similar 
problems in their own projects. 

a. 
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procedures were tne outcome of several different influences 
ranging from defects inherent in the establishment of the 
project to inexperience in curriculum and instructional 
design.^ It would be a difficult task and of little use to 
recount the various attempts made at Chanute to bring some 
o;*der to the production of lessons. The task would be dif- 
ficult because, as with the implementation p-lans, several 
different procedures were tentatively adopted and of little 
value since the .administration of these procedures was 
ineffective. 

The most instructive view of these practices would be 
that which sees them as a natural outcome of deficiencies in 

-the establishment and management of the project. Thia view 
will be adopted here. No attempt, therefore, will "be made 
to describe completely the development of a CBE lesson from, 

.the planning stages to its operational use or to enumerate 
the number of different lesson development procedures , * 
; adopted. Unless otherwise noted, any specific mention of 
procedures will refer to those employed during the tenure of 
the' last TRAB project officer who managed the service test 
for the major portion of time before the project review in" 
May, 1974. 

During the phase cf the Chanute service test under 
discussion, there were no timelines imposed on lesson devel- 
opment by the project management. Rather, each author 
worked at his own speed on a PLATO lesson until it reached a 
state of perfection that, was satisfactory to him. With no 
timelines to be met, lesson development was quite slow. A 
factor lengthening development time was the fact that there 

, were no precisely stated or enforced procedures for lesson 
development. Thus, the authors, became the sole judges of 

. when their lessons were ready for classroom use. The policy 
of allowing the authors ^to determine when their lessons are 
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dbmpleted is particularly unfortunate in the early stages of 
a project. Novice authors frequently tend to be overly con- 
cerned with cosmetic and generally nonessential aspects of 
their lessons when they are not constrained by timelines for 
lesson production. Without such timelines, an author's 
boredom is often the major criterion for when a lesson is 
completed. 

Such was the case at Chanute nhere it was thought to be 
useful "to determine how long it would take novice program- 
mers and lesson designers to program one student contact 
hour (Green, 1973, p^. 27). "For the project's first lessons, 
each author was asked to select a tppi-c from a given portion 
of the GPVC for a PLATO lesson a <d then to begin the task of 
designing and programing the lesson. Working independently, 
the authors took an average of 650 hours to complete one 
student contact hour of CBE material (Green et al . , 1973,* 
p- 38-j). Whether or not this figure includes the time 
needed for trial student runs and lesson revision is not 
clear. If it does not, the actual time needed to ready one 
student contact hour by the Chanute PLATO authors may have 
been as high, as 700 hours. 

Even for a project's first lessons, 650 hours of 
authoring time for one student contact hour is an unusually 
slow production rate. The Aberdeen Proving Grounds required 
about 400 hours per student Contact hour for their first 
lessons. Aberdeen's first lessons were produ6ed under simi- 
lar circumstances to those of Chanute with each author 
being a CBE and instructional design novice and undertaking 
virtually all of a lesson's development (Himwich, 1977). 

Related to the long lesson development times experience 
during the first phases of the project was the fact that 
there were no established lesson structures. > Instead, since 
the project officer perceived the major objective of the 
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test to be research, practically each lesson became an 
experiment in instructional design* These experiments were 
frequently designed by the individual author with only the 
approval of the management/ evaluation staff which was no 
more experienced in the area of instructional design than 
the author. Thus, with one lesson designed primar ily to 
test the PLATO system's graphics capabilities, another to 
experiment with simulation, another to test the effective- 
ness of microfiche, etc., the CBE courseware took on a some- 
what random appearance. This randomness extended not only 
to the the basic intent of the lesson but also the conven- 
tions employed by each lesson for proceeding through that 
lesson and the requirements for its successful completion by 
the student. Thus, the first lessons taken as a whole did 
^ not have an appearance of belonging to the same course. 

Although these remarks have been made in reference to 
the project's first lessons, they apply in general to all 
the lessons produced during the period of time under discus- 
sioni For the .most part, each lesson was designed and 
programmed to test some aspect of CBE that intrigued its ' 
author. The PLATO authors were allowed to take the initia- 
tive in choosing the topics of their lessons from the GPVC 
material, in determining what capability of CBE was to be 
tested by their lesson, and in designing the lesson to meet 
the experimental objectives. The role of the management/ 
evaluation staff during this time was primarily advisory. 
Their advice apparently did not include guidelines that 
would have given the lessons the appearance of belonging to 
the same project or timelines for how long a given lesson 
should be worked on. 

That the authors worked in such a permissive atmosphere 
appears to be primarily the result of two conditions. The 
first of these, and probably the most important, was the 
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absence of service test obj.ectives more precise than a gen- 
eral charter to research aspects of CBE. The second was the 
lack of experience on the part of the management/evaluation 
.staff in the crucial areas of curriculum design, educational 
research, and management. With vague objectives and little 
of the expertise needed to direct a research-^oriented cur- 
riculum development project, a too-heavy burden was put on 
the management/evaluation staff of the service test. 

The authors had become very used to ^working indepen- 
dently early in the project. If there were any attempts to 
impose order on the lesson development processes such 
'attempts would naturally be met with resistance. In an 
interview, management/evaluation staff expressed the view 
that the PLATO authors expected some justification for any 
change in procedure they were asked to make. For such 
authority, the management/evaluation staff had neither an 
implementation plan, nor relevant experience, nor precedent 
to offer. As. a result, their attempts for an active manage- 
nient of the project *s lesson production were for the most 
part ineffectual. 

Summary 

There is no known record of the output of the Chanute 
PLATO service test during its first 21 months. Several 
repor':s indicate the progress at various tim^s before May, 
1974, but no summative report exists. Projecting such data 
from the first annual report (Green, 1973), suggests that 
the first phas.e of the service test produced about 22 stu- 
dent contact hourn CBE instruction. ^ Allowing 2 months 



^The estimate of the total number of student contact 
hours is derived by projecting the ratio' of the total number 
of manhours available with the total number of student con- 
tact hours produced as of May, 1973* 
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Of training time for the eight' authors the rate of CBE les- 
son production was about 112D manhours for each student con- 
tact hour produced. 7 Also, 80 students were, used in testing 
these lessons as part of their formative develop)ment . There 
is no record of any of these lessons being used"" without mod- 
ification as mainline instructional materials at Chanute or 
as part of structured, research . However, '16 of the lessons 
were modified to he consistent with the objectives of the 
second phase and were included among the 33 PLATO modules 
used in that phase. 

This estimate of the productivity of tho first period 
of Chanute' s service test roughly agrees with those of some 
service test staff and those of outside programming and edu- 
cational consultants. Even by doubling these output rates, 
the productivity would have to be considered poor. In dis- 
missing the management, several of the factors contributing 
tOvthis poor record have been mentioned. In summary, these 
factors were: 

\1) the absence of precisely articulated pro- 
\ jept objectives, 

2) \the "attempt to maintain a simultaneous 
•research and operational orientation to 
the project, 

3) uncertain designation of leadership fol- 
lowed by inexperienced leadership, 

^) the absence of structured CBE lesson pro- 
duction proc^idures, 

5) slow hardware acquisition and PLATO system 
. unreliability during the first six- months 
of the service test. 



7lt is interesting to note that the report from which 
these numbers are derived gives 445 manhours 'as the time to 
develop one student contact hou»^» Thus, as a rough esti- 
mate, the authors were spending more than half of their 
'time in activities not directly related to CBE lesson pro- 
duction. 
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Although no significance is intended in the order of presen- 
tation of these factors, interviews with man'^j^ement person- 
nel have indicated a"strong feeling that' the first was the 
major contributor to thf service test's lack of productivity 
during this time. While this -factor is surely important in 
this respect, it seems clear that all five of these factors 
contributed synergist ically to the serv ice test • s lack of 
productivity. Nevertheless, the presence of any one of 
these would have severely diminished any project's chances 
for success. 

Final responsibility for the absence of useful activ- 
ity during the test's first phase must rest with AFHRL and 
ATC. Had th^ey pursued more -actively, the tasks of settling 
on and supporting an implementation plan early in the ser- 
vice test a substantial portion of the 18 month first phase 
might have been salvaged. Such a course would have contri- 
bute^ to a- positive esprit de corps by showing high bureau- 
cratic interest in addition to giving the project a sense of 
direction which by itself might have boosted morale. Simi- 
larly, ATC and AFHRL were in a position to insure that the 
management/evaluation staff was sufficiently experienced to 
■carry out the difficult task of administrating and' monitor- 
ing the test. In May, 1974, when ATC and AFHRL finally 
determined to closely examine the test and take a more 
active hand in it, the test was put under experienced 
leadership and entered its most productive period. 



52 



45 



ISD MANAGEMENT OF THE SERVICE TEST. < j 

Early in '197^*, the problems of the CJ|anute. service test 
..came to the attention of the agencies^ which were materially 
concerned with the test's outcome. A> formal review at 
Chanute AFB was convened ' on 'May 15, 1971', .with reprensenta- 
tives from ATC, AFHRL,.ARPA, the Educatiiinal Testing Ser- 
vice, and GERL attending.' The presentation offered by the 
service test personnel reflected accurately the fact ttiat ' 
the test had been up to that time a dlsorgani-ied' effort and ' 
that little useful information about CBE had been gathered. 
As a result .the service, .test was put under new management 
and a 'new research agreement was drawn up. 

The new research agreement called for the Department of 
Weapons ^Systems Support Training (WSST) Instructional 
Systems Develo'pinent (ISD) chief to take over the management 
of the PLATO authors from TRAB. Th is ' ind ivid^Jal was already 
revising the Special Purpose Vehicle Maintenance Courses 
(SPVO.S It was felt that by using .the PLATO system to pre* 
sent the SPVC as revised by J^he ISD effort a measure of 
the "true costs and' capabilities of PLATO IV in a systems 
approach to Air 'Force technical training" could be obtained 
(AFHRL, 1974). 

The new research agreement called for the ISD team to 
integrate PLATO .lessons into the SPVC: These lessons would 
then be offered >/ith the other revised components of the 
course as operational trailing materials. The management 
plan for the ISD effopt states -explicitly that "PLATO 
lesson/learning strategies and methodologies will be limited 



^Like the GPVC , the SPVC had several" informal verbal ' 
designations. The official name of this sequence of courses 
is 3ABRi*7231-1/A/B/C. 
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to presently 'available, hardware and programming techniques" 
(Gissing,- ig?'*). 'TKus, rather than regard the PLATO system 
as a research tool, in the new phase of the service test it 
would be used alongside other instructional media in an on- 
going course . ^ . ' 

The management plan called for the development of mate-; 
rials to support 88 training objectives, or these, 50 ' 
objectives were knowledge oriented, and 33 of the knowledge 
oriented objectives' were judged to be suitable for -PLATO 
support. Happily-, the SP„VC and GPVC were, not totally dis- 
similar in their content. ' Wi'th some modifications, les-sons 
from the first phase of the service ^test could be used to 
support 17 of the 33 objectives. QBE lessons for the 
remaining 16 objectives were completed in., the time allotted 
by the management plan. Along with the modified I'essons 
from the first phase of the service test these PLATO lessons 
became part of the SPVC in January, 1975. 

* 

Comparison of, the First and Second Phases 

Because it was the only phase of the service .test that 
produced a substantial amount of analyzable data on C3E use 
in military trailing, the second phase has been extensively 
documented in reports by the Air Force and CERL. To report 
on the same aspects of the second phase in the detail that 
was lavished on the first would largely duplicate what has 
already been written. For this reason, a bibliography of 
these reports with a brief description of their contents is 
included in this section. To supplement these reports and 
to continue with the discussion of the entire service test, 
a comparison of important aspects of the first two phases 
will be drawn. 

An important difference between the first and second 
phases of the Chanute service test was that the second 
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phase's goals and means of attaining them were carefully 
articulated at the outset in an implementation plan.'Tfiis 
plan laid down the responsibilities of all of the parties 
involved with the proj.ect. In' particular , the sponsi.bil^ 
ity for managing th.e eight"" PLAnTO authors was ^ y,.., to the 
ISD chief fthile the role of TRAB was relegated to that of-, 
li^son for the service test with „ARPA, ATC , AFHRL, *and 
CERl. TRAB wa5 'also given the responsibility for conducting 
tours of the PLATO project .so that there would be no inter- 
ference with the PLATO authbrs' worl<.' For the e.valuation 
'activities that were the joint responsibility of the AFHRL 
representative and TRAB, the -plan specified that guidelines 
were to be drawn up which.. -would preclude the" evaluation 
procedures from interfering .with the lesson development. - • 

By simply possessing a management plan, the" ISD phase 
of the service test differed substantially from the first 
phase. In delineating resppnsibil-ities of the various 
agencies involved in the test, the distin^ion from the 
first phase was still more striking. Added to these differ- 
ences was the fact that the management plari set out time- 
lines for PLATO lesson production and valijdatVon. Thus, 
the new manager of the service test possessed/ some standard 
by v^jhich bo govern the rate of lesson .production. In fact, 
knowing the amount of • lesson ^material needed and the amount 
ai^feac|^ existing, he initially established production rates 
, of 50 hours per student contact hour. Being based on the ' 
production rates for programmed texts, this expectation was, 
perhaps, oVerly optimis'tic. Ne.vertheless , this goal did- 
establish an objective that could be articulated to th'e 
authors and. could be used as a touchstone to determine .pro- 
gress. Without such a management device, attempts at 
administering a curriculum development effort are likely to 
be ineffectual, as was demonstrated in the first phase. 
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During the first phase, the administration of tne ser- 
vice test was placed in hands inexperienced in. both curricu- 
lum development and sound management practices. ' The new 
manager, as ISD chief, was r.esponsible for revising 107 
existing courses to improve instructional efficiency while 
at least maintaining student performance. When he undertook 
to manage the PLATO service test, he had already carried out 
revision of about 32 of these courses. In a total of 22 
years at the School of Applied Aero-space Sciences, he had 
occupied a variety of positions from instructor to WSST ISD 
team chief. He had spent seven years with the ISD team and 
two years as its chief. Thus, he possessed extensive 
experience in the designing and administration , of curriculum 
development projects. 

Another striking difference between the first two 
phases of the service test was the manner in which curricu- . 
lum and lesson design was carried out. During the first 
phase, ea~ctr-PLATO author carried out for himself most of the 
ac^ivities needed, in CBE lesson production. These activi- 
ties included lesson design, TUTOR programming, providing 
subject matter knowledge, and lesson validation. These 
raquirements, difficult to meet by even the most experienced 
educators, were beyond the qualifications of the PLATO 
authors. As part of an ISD project, however, the authors 
joined a more integra'ted and disciplined effort with defined 
roles and procedures. While they, in fact, designed most of 
the CBE lessons, their primary responsibility, in the view 
oT the ISD team chief, was to provide TUTOR programming 
expertise. The details of the ISD curriculum design model 
are given in several of the reports summarized after the 
discussion of the ISD phase. 

Under ISD management, the service testes orientation 
was- very different from what it had been under TRAB 
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management. Whereas before the test was attempting to 
research basic CBE capabilities, the second phase was simply 
motivated to develop training materials that would be used 
as part of operational training. Being operationally 
oriented did not permit the luxury of selecting the most 
interesting topics to teach via the PLATO system or of 
determining experimentally how those topics might be most 
effectively taught. In practical terms, the shift to the 
production of CBE materials determined that the PLATO 
authors could no longer play a major role in directing their 
own work. 

Under the new management, CfiE lesson production 
improved markedly. While no good records were kept, the ISD 
chief estimates lesson production at 100 manhours per stu- 
dent contact hour. Keeping in mind that the PLATO authors 
produced as much CBE lesson material in six months under the 
ISD management as they had produced in over 18 months under 
TRAB management, it is- clear that this estimate is at least 
♦plausible. The reader must be warned, however, that these 
estimates are valid only if taken as general comparisons in 
productivity between the two phases; no careful documenta- 
tion of lesson production rates for either phase is 
available. 

Finally, the ISD effort has left at 'chanute AFB a tan- 
gible product. The CBE lessons altered from the first 
phase's work or developed during the second phase are still 
being used in operational training at Chanute. Since Janu- 
ary, 1975, the CBE lessons required for the SPVC have been 
used by an average of 350 students per year for approxi- 
mately 7000 student contact hours. By itself, this outcome 
is praiseworthy for its contribution to military training as 
well as for the accumulation of data giving evaluators some 
insight to the potentials and problems of CBE use in 
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military technical training. In comparison to the first 
phase, this accomplishment' is all the more impressive for 
its magnitude and the efficiency of its execution. 

Reports Describing the ISP Phase of the Service Test 

The following reports and articles discuss various 
aspects of the Chanute service test from May, 1974, to, 
April, 1975. Here, the title, author and a brief descrip- 
tion of the contents of the document are described. Full 
bibliographical information for each of these publications 
may be found in the List of References. 

• An Overview of Chanute Lessons by J. A. Klecka (1977a). 
An analysis and summary of eight PLATO lessons written 
during th-e ISO phase of the service test. The ^study is 
based on in-depth reviews of these lessons both in the form- 
ative and final stages of development. An indication of 
general characteristics of ISD PLATO lesssons is given. The 
study concludes with an assessment of the instructional 
effectiveness and utilization of the instructional capabili- 
ties of these lessons. 

Three Aspects of PLATO Use at Chanute AFB by J. A. 
Klecka (1977b). The fi^st of these chapters describes the 
implementation of an ISD systems-design, team-authored PLATO 
lesson production prbcess; the second, the use of the PLATO 
system for data management, student testing, and other 
related computer managed instructional uses; the last exam- 
ines the changes made to the eight PLATO lessons sampled in 
Klecka (1977a) in order to achieve the validation criterion. 

. Evaluation of PLATO IV in Vehicle Maintenance Training 
by B. E. Dallman, P. J. Deleo, P. S. Main, G. C. Gillman 
(1977). While nominally covering the entire Service Test, 
this report of about 3OO pages deals mostly with the ISD 
phase. The data accumulated during this phase is presented 
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and analy.zed. Additionally, extensive anecdotal information 
concerning this phase is distributed throughout the report. 
To obtain a balanced view of much of the subject matter of 
this r-eport, its reading should be accompanied by the read- 
ing of Misselt et al. (1977). 

"A Critique of the Final Evaluation Report of the 
\Computer-based Education Service Test at Chanute AFB" by 
A. L.. Misselt, H. A. Himwich, L. D. Francis, R, A. Avner, 
K. K. Tatsuoka, and J. A. Kleaka (1977). The" critique is a 
close chapter-by-chapter commentary on Dallman e€ al. 
. (1977). This commentary was written by CERL personnel who 
acted as evaluation, instructional, and programming consul- 
tants for the Chanute CBE service test. 

"Military Instructor Attitudes Toward Coraputer-Based 
Technical Training" by j; A. Klecka (1977c). The attitudes 
of Chanute instructors toward the PLATO system in vehicle 
maintenance training were investigated. If was believed 
that insti;uctor atttitudes would influence instructor and ■ 
student acceptance of CBE in the training environment. The 
attitudes were measured via, surveys and interviews with the 
instructor staff and other related personnel at Chanute over 
fts^^ period of several months. The results gave a basically 
positive picture of instructor attitudes and their conse- 
quent impact on student interaction with the PLATO IV 
system . 

"Att-^tudes and Performance of Military Students in 
Computer-Based TecHnical Training" by K. K. Tatsuoka, A. L. 
Misselt, and P. L. Maritz (1977). The student attitude data 
reported by Dallman et al.(1977) are re-analyzed and com- 
pared with measures of performance. The results and inter- 
pretations differ somewhat from' those geported" by -Dallman 
et al . 
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Summary 

The ISD phase of the Chanute service test was in many 
respects a new beginning for the test. Although the staff 
of PDATO authors remained the same,va new research agreement 
initiated the phase along with new management, new objec- 
tives and a new approach to CBE lesson development. The 
result was an eight-month effort which was more productive 
in terms of CBE lesson material written and validated,. CBE 
materials implemented in technical training, and information 
concerning CBE use in military training than the rest of the 
four year service test combined. That Dallman et al.(1977) 
frequently treat the ISD phase of the service test as the 
service test and devote substantially more than half of the 
total pages of the final report to it may be taken as a 
token ofvthe relative importance of the ISD phase. 

As Klecka ( 1 977a )4 points out, the CBE lessons developed 
during the ISD phase were not exemplary in terms of using 
the -PLATO medium in art effective manner. This fact may be 
in part accounted for by the fact that, despite extensive 
experience in curriculum development in more conventional 
media, the ISD team had little experience with CBE. More- 
over, the time constraints that the ISD team were under dic- 
tated that elaborate, media-sensitive PLATO lessons were an 
unessential luxury. Finally, in the development of any 
instructional materials, the ISD team endeavored to design 
the materials to teaqh only what was required and no more. 
'The fact that the ISD PLATO lessons did not represent opti- 
mal uses of CBE. does not greatly detract from the achieve- 
ment of actually producing a large body of instructional 
material . This accomplishment, substantial in itsel f , is 
the more impressive within th« context of the rest of the 
service test. 
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Although it would be perhaps impossible to attribute 
the relative success of the ISD phase to any one particular 
cause, it is plausible that one or more of the differences 
between the TRAB and ISD phases of the project was the 
source. The major differences are summarized below. 

1) An implementation plan governed the ISD 
phase. The plan, which had the- approval 
of ATC ana AFHRL, precisely articulated 
objectives and means and timelines for 
achieving them. The TRAB phase, on account 
of an inability to obtain agreement from 
ATC and AFHRL, possessed no finally- 
approved implementation plan. 

2) The PLATO authors were directed by an 
experienced curriculum developer during 
the ISD phase; during the TRAB phase, the 
managers were novices in curriculum devel- 
opment, CBE, and management. Again this 
would seem to be due to casual interest in 
the service test from AFHRL and ATC. 

3) During the ISD phase, CBE materials were 
developed in accordance with validated 
instructional development procedures. 

In the ISD development procedures the 
PLATO authors filled an important but 
precisely circumscribed role. Under TRAB 
management, the PLATO authors were respon- 
sible for virtually every facet of CBE 
lesson development. Moreover, lesson 
development procedures were left to the 
discretion of individual authors. 
That these differences existed between the two phases 

is a matter of fact. That one or more of them could explain 
the striking difference in outcomes between the two phases 
appears to be a matter of common sense. In addition to 
common sense, interviews with service test personnel bolster 
the conclusion that these factors were irtiportant in deter- 
mining the relative success or failure of each phase. 
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Foreword 

The following commenfe^s... partially fulfill a contractual 
obligation between the Defense Advanced Research Projects 
Agency (ARPA) and the Computer-based Education Research Lab 
oratory of the University pf Tllinois (CERL) to provide a 
critique of reports produced at the various PLATO sites sup 
ported by ARPA. The comments supplement the communications 
exchanged between Chanute AFB and CERL during the course of 
a long and close working relationship. They are intended y 
give the reader a view of the project in the context of th4 
general problem of implementation of new instructional tech 
nology as well as providing independent comments on histori 
cal and procedural matters. The comments are based on the 
experience of the CERL staff in aidinjg the implementation 
and'operation of over 20 military Computer Based Education 
(CBE) sites, as well as professional expertise in the areas 
of military instruction, instructional design, and evalua- 
tion. 

We were able to^prepare this critique- because the Air 
Force Human Resources Laboratory (AFHRL) staff kindly made 
their report available to us in draft form. Unfortunately, 
contract completion dates are such that publication of this 
critique must be made before the final version of the AFHRL 
report (Dallman, DeLeo, Main, & Gillman, 1977) will be 
available. While knowing that the comments we and other 
reviewers have made*may affect the final contents of the 
AFHRL report, time requirements have forced us to base our 
remarks on the. draft version available in April, 1977. 
Therefore, readers may expect to find that some problems or 
questions which our critique raises have been corrected or 
answered in the final version. 
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The organization of this critique parallels that of the 
report; pothments are offered section-by-section. In writing 
the critique, we have made the assumption that the reader has 
a copy of the AFHRL report in hand in order to follow the 
commentary; we have provided only a brief restatement of the 
procedure's and findings on each topic. To facilitate refer- 
ences to the AFHRL report, the term "chapter" is used to 
denote the major se,ctions identified by Roman numerals. 

We have found the AFHRL report to be well-organized and 
informative. It is an important contribution to the liter- 
ature on CBE and on the implementation of curricular innova- 
tions. All of those who assisted in the implementation and 
reporting of the Chanute CBE project have shown commendable 
effort and prof essiona.l expertise in meeting the problems of 
testing new technology in a realistic training environment. 
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I. INTRODUCTION 

Project Genesis 

The section titled ''Project Genesis" is a brief but 
accurate summary of the origins of the service test at 
Chanute. It should be noted, however, that in addition to 
stating the broad objectives for the service test, the ARPA/ 
AFHRL/ATC research agreement (cited in this section) also 
outlined an implementation plan for achieving those objec- 
tives. It called for the development of computer-based 
instructional materials on the PLATO system which would 
duplicate the content of Chanute's General Purpose Vehicle 
Course (GPVC). With identical content for that course avail 
able in both computer-based and conventionally-delivered 
modes, it would have been possible to conduct a comparison 
of the two media categories. Under that plan the conven- 
tional course would have provide'd a standard against which 
the "tiost effectiveness, acceptance, Incorporability (into 
ongoing technical training), and reliability" could be inves 
tigated. For reasons described later in the report and in 
this critique, that initial plan prove.d to be infeasible. 

Responsibil ities 

The delineation of responsibilities for AFHRL and ARPA 
provides a concise description of their roles. However, the 
role of the Training Research Applications Branch (TRAB) is 
characterized in the original Research Training Agreement as 
the 

"Technical School OPR [Office of Primary 
Responsibility] for coordination of the 
experimental service test with ATC/XPT, AFHRL, 
ARPA, and the University of Illinois. This 
office will monitor the research program by 
developing a programmed action directive for the 
duration of the three-year research agreement." 
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While the full intent of this statement is not altogether 
clear, it seems to imply a somewhat larger role than the 
responsibility ascribed to the TRAB in Chapter I, i.e., "to 
evaluate'^and document the project outcomes". 

The difficulty in describing the TRAB's role is that 
the Chanute service test actually consisted of three distinct 
"projects", which are referred to throughout the report as 
"phases".' Since each phase had objectives and management 
unlike that of the others, it is possible to think of them as 
different projects. The responsibilities outlined in this 
section of Chapter I. are true of the second project or Phase 
II;"^they dp not necessarily describe the roles taken by the 
various agencies and units (especially that of the TRAB) 
during the other phases. 

Also, while it is clear that it was ATC's responsibility 
to ^ovid^ manpower and facilities, it is apparent from the 
discussiomin Chapter II under the heading "Selecting an 
Evaluation Paradigm" (as well as the experience of CERL) that 
ATC had a larger policy-making role. 

Project Personnel 

Except that it fails to mention the participation of 
members of the instructional systems development (ISD) team 
during Phase II, the description of the service irest staffing 
is correct. All of the personnel described in this section 
did contribute to the project, but they were not all present 
simultaneously or throughout its duration. 

Scope of Development 

The development described in this section was that of 
Phase II of the service test. Phases I and III are described 
separately in Chapter II. 
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II. BACKGROUND 

P LATO System Description 

The desc-iption of the PLATO system is brief, but accu- 
rate\ Readers wishing a more complete background may refer 
to the article by Smith and Sherwood (1976) or the bibliogra- 
phy by Lynran (1977) . ^ 

HistoricaJ Perspective 

P'j£se__I. The- discussion of the events of Phase I is a 
highly abbreviated description of the activities and deci- . 
sions of this period. Some additional information is " 
provided in the following comments. 

The 'first implementation plan for the service test (con^ 
tained in the ARPA/ AFHRL/ATC research agreement) called for 
the comparison of^CBE instructional materials with conven- 
tional materials already in use in the GPVC. Hence the 
personnel expected to develop the QBE materials, i.e., the 
authors, were selected from among "^th'e GPVC instructors. As 
indicated In the AFHi?L report, the eight individuals who were 
selected y-&rie"d widely in the extent of their teaching exper- 
ience and Knowledge of the subject matter to be taught, and 
they were uniformly inexperienced with regard to development 
of CBE materials. They received their initial training in 
the use of the TUTOR language at CERL in the Fall of 1972. 
The TUTOR workshop they atte'nded was the first of several 
conducted by the Military Training Centers (MTC) group of 
CERL for the ARPA-Pup'ported PLATO projects. For a descrip- 
tion and evaluation of the MTC TUTOR course, see the report 
by Francis (1976b). 

As indicated by the report authors, the lesson develop- 
ment efforts during Phase I were exploratory and served to 
allow the Chanute authors to expand their skills and develop 
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their interests. Compared to Phase II lessons, those written 
"In Phase I exploited system capabilities much further in 
terms of their wider use of computer-generated questions, 
individualization via feedback and branching, and animated 
graphics. Each of the Phase I lessons was considered by"the^ 
staff to be a mini-experiment in CBE; each was thought to 
investigate the use of one or more of the features described 
above. Developed by persons with little^or^— no- experience in 
instructional design or CBE, these lessons were written with 
idiosyncratic instructional strategies and often failed to 
represent the educational concept or CBE feature they were 
designed to investigate. If the reader keeps in mind the 
statement in the report that "Each author became an investi- 
gator in his own right", and the fact that these authors were 
highly inexperienced, one can easily construct an image of 
the resulting lessons. Nevertheless, experience was gained 
in the areas of lesson design and programming and a number 
of lessons were developed, many of which were later modified 
for use in operational training during Phase II. 

There were a number of problems and frustrations during 
Phase I which hampered the productivity of the project. In 
addition to the constraints mentioned by the report authors 
(e.g., delays in equipment delivery, computer memory short- 
ages, system unreliability, changes in and lack of documen- 
tation of the TUTOR language), there were several management 
and ^environmental problems which interfered with the goals 
of the service test. These included the ±Z2k of commitment 
on the part of certain operationally-oriented personnel to 
the test's exploratory nature, turnover and inexperience 
among management personnel, and the lack of an approved 
implementation plan. These .latter problems are discussed in 
detail in the chapter by Himwich (1977a). 
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By early 197^ it was apparent that the CBE lessons 
developed to replace selected classroom lectures in the GPVC 
did not represent the most effective application of the PLATO 
medium. This, plus the fact that student flow through the 
GPVC was to drop drastically, led to a suggestion that CBE 
be implemented in the common area- of four Special Purpose 
Vehicle Courses (SPVC). Not only was it hoped that the - 
existing GPVC lessons could be utilized in the SPVC, it would 
also be possible to take advantage of the larger student flow 
for a full program of lesson validation and program evalua- 
tion. Fij|^«4^y, since the SPV courses were about to undergo 
a systematic Revision," it would be a simple matter to inte- 
grate PLATO as a primary instructional medium, hence simpli- 
fying many implementation and evaluation problems. 

The suggestion to shift CBE development to the SPVC 
was put in the form of a proposal and submitted to ATC and 
AFHRL for approval. This approval was granted following a 
formal project review in May 1974. However, more than just - 
a shift in target courses was decided. There was also a 
shift in the general orientation of the project from an 
exploratory research focus to one of integrating PLATO into 
operational training. This was to be accomplished by .assign- 
ing the authoring staff to the supervision of the 3340th 
Technical Training Group'? ISD team. The May 1974 project 
review marked the transition from Phase I to Phase II and 
hence was an important turning point in the test ' s* hi story . 

Additional data on Phase I, the first 22 month-s of the 
service test, would have been useful. Some data are pre- 
sented in the chapter by Himwich (1977a). Also, several pro- 
ject reports exist which supply some of these data. The 
most useful of these is the report by Green (1973). It gives 
a detailed description of the authors selected, an extended 
discussion of early management problems of the service test 
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and a summary of the early lesson development efforts of the 
authors. An interesting contribution of Green's report is 
its tracking of /author attitudes and morale through early 
sei 'ice test events. 

Phase II . This section gives a brief overview of som§ 
of the activities and outcomes of the service test's second 
phase. The brevity i's justified because most of the discus- 
sion in the remainder of the AFHRL report is devoted to data 
collected' during' Phase II. 

Phase III . A thir<l phase of the test was originally 
conceived as a way to utilize surplus authoring time ayail- 
able during the validation period for the Phase II lessons; 
The contract period with ARPA ended on September 30, 1975, 
but was extended until June 30, 197.6, to allow time for fur- 
ther analysis of Phase II evaluation data and the completion 
of the Phase III studies. The revised management plan under 
which Phase III was begun was thought by CBE experts from 
CERL to be overly ambitious though feasible in its main 
intent. The management of this portion of the project was 
as unstructured as that of Phase I, but the' exodus of -trained 
PLATO authors was itself sufficient to make the objectives 
Unattainable. A detailed summary of the staffing problems 
of the service test may be found in the chapter by Himwich 
(1977a). 

Although never officially described as such, a fourth 
service test phase was funded by ATC* Running from 
July 1, 1976, to September 30, 1976, it was intended to 
al^ow additional time to collect data on the cost saving 
contributions of the PLATO system to the operation of the 
SPVC. Specifically, the goals of this follow-pn effort were- 
to determine the time savings due dfrectly to PLATO (i.ej, 
above and beyond those due to group-pacing), to determine 
the benefit from on-line testing, and to determine the max- 
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imum student load that could be handled with Chrnute's com- 
plement of 30 PLATO IV terminals. To answer this last ques- 
tion, the SPVC lessons were used as replacements for a number 
of the lectures in the GPVC in order to increase the total 
student flow through the materials. The results of these 
"Phase IV" studies are reported In an unpublished manuscript 
prepared by the former chief of Chanute's Training Research 
Applications Branch. These results are not discussed in the 
AFHRL report. 

Instructional Setting 

The "instructio'i.x setting" section of the report deals 
with the relationship among the four target courses, the 
manner in which the courses were changed as a result of the 
ISD team's redesign of the curriculum, the basic student 
performance indices that were measured, and the general 
nat.ure of the material selected for CBE development. This 
information is fundamental to an understanding of the Phase 
II effort and, considering the small amount -of space 
allotted, it is presented with admirable clarity. To facil- 
itate interpretation of later sections, a few of the more 
important points deserve special attention. They are high- 
lighted in the following comments. 

One of the most basic changes resulting from the ISD 
redesign of the courses is that classes progressed through 
the revised versions at the rate of their slowest members 
(i.e., group-pacing was used). If a class were heterogeneous 
in previous background and learning rate (the usual case 
since homogeneous grouping was not attempted), the faster 
students were assigned supplemental material to keep thera^ 
occupied. Thus, the change to group-pacing allowed more 
flexible scheduling than was possible in the conventional 
courses, but it did not pVovide optimum pacing for all stu- 
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dents. An individually self-paced system would have been 
ideal, but according to interviews with Chanute AFB person- * 
nel, such an approach could not be implemented because of a 
shortage of laboratory equipment and because the bulk of the 
instruction in the specialty shreds was presented via lec- 
tures. To develop individualized instruction to replace the 
lectures was thought to have required more effort tjian could 
be justified by the relatively small student flow through the 
individual shreds and the expense of the additional laboratory 
equipment that would be needed. 

Another change affecting the instructional efficiency of 
the revised common course segment relative to that of the 
conventional version was the introduction of the "Block X" 
concept. The effective length of the courses was shortened 
by assigning a number of topics to be studied individually as 
homework. It is our understanding that much of this material 
had been presented during class time for the "conventional" 
common course segment. In comparing the two ver.^ions, 
therefore, it should be noted that the revised segment had 
Tess content than did the conventional version. 

The reversal in proportions of ti^e spent on "knowledge" 
and laboratory tasks was due both to decreased emphasis on 
the knowledge areas and an increased emphasis on the perform- 
ance skills taught in the courses. These shifts in emphasis 
came as a result of the job analysis conducted by the ISD 
team as the initial step in development. A second factor was 
that the lock-step pacing in the conventional version had 
allowed more "slack" in the scheduling of the knowledge areas 
than in the laboratory activities. Hence, there was more 
time to be saved through group-pacing in the cognitive por- 
t ions . 

Another item that should be noted is that the content of 

r 

the block exams for the revised common course segment dif- 
fered from those used in the conventional version. Because 
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of shifting emphases in the content of the first four blocks 
of the courses, the block .exams usecf in the conventional ver- 
sion were not valid for use in the revised segment. It is 
not possible, therefore, to compare directly the bl<3ck exam 
performance between students taking the conventional and 
revised versions. Although the tests generally measure 
similar topics, their coverage is sufficiently different to 
keep them from being parallel. 

This section of the AFHRL report gives only a brief 
description of the CBE lessons produced during Phase II. A 
more thorough presenta'tion is available in the report by 
Klecka (1977a) which gives a detailed discussion of eight 
Phase II lessons selected by an AFHRL evaluator. Each les- " 
son was reviewed in depth and the resulting comments were 
synthesized to give an overall description and assessment 
of the lessons reviewed. The Klecka report was used as a 
resource and database for Chapter VI of the AFHRL report. 

Selecting an Evaluation Paradigm 

It is commendable that those who planned the AFHRL eval- 
uation systematically attempted to collect data on the infor- 
mation needs and priorities of the several agencies connected 
with the Chanute service test (i.e., the variojs audiences 
for the e>)aluation report). The result of this effort was 
the assigning of highest priority to those areas of greatest 
interest tc^ ATC , the agency which would have to make deci- 
sions regarding support for future applications of CBE in 
Air Force training. 

In view of ATC's requirements for data on which to base 
decisions, the evaluators chose to conduct a comparative 
evaluation rather than a more generalizable " evaluation-as- 
research" approach as might be suggested by Cronbach (1963). 
A comparative evaluation usually seeks to provide data for 
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selection of one of the specific products being evaluated, 
but confounding of subject-matter, student population, and ^ 
instructional desigji variables with the media used for 
instruction would limit the generalizability of a single 
comparative study. Recognizing these problems, the authors 
of the AFHRL report have made an effort to identify them for 
the reader and to provide sufficient information- so that 
their findings can be combined with results obtained else- 
where. By providing ample description of the constr.aints 
under which the data were collected, the authors enable 
interested readers to draw upon these results and others to 
state generalizations and hypotheses for. further investiga- 
tion. 

Alth-^ugh no explicit rationale is given for includ- 
ing the summary of CBE effectiveness studies, it is apparent 
that one purpose is to highlight an issue in evaluation 
design found in several of the studies cited. Namely, when 
CBE or other self-paced media are compared with conventigrtal 
classroom presentations, the time savings typically enjoirecJ- 
by the CBE groups can not be attributed entirely to the 
effects'of CBE- Such savings might be due merely to .the 
fact that self-pacing or group-pactng has been employed 
rather than to any unique CBE characteristics. Since CBK was 
to/be compared with a convent ional ■ lock-stepped presentation 



selection of an evaluation paradigm. Two studies were cited 
in which this problem was solved by com'paring the CBE mater- 
ials with comparable materials presented by other self-paced 
media. This type of control was attempted for a^ few lessons 

in the Chanute service te^t (see section on PLATO system 

* 

effectiveness in Chapter III), but the lack of non-CBE (i.e., 
off-line) self-paced materials made it impossible to imple- 
ment on a larger scale. 



format in the Chanute service test, the question of ho,w to 
isolate CBE's unique contribution was 'an important one in the 
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Another apparent reason for citing CBE effectiveness 
studies wa^s to summarize their findings as background . for 
the re^^ts obtained in the Chanute service test. *A study 
of par^-cular interest wajs that conducted by the U. S. Army 
Ordnance Center and School *(Dept of the Army, 1975). It 
involved a comparison of the time required to achieve a pre- 
. set criferion of mastery for eadsji^ of a series of CBE lessons 
and for a parallel series of lessons presented on conven- 

^ tional self-paced media. The project staff reported time 
savings in favor of CDE in most of the lesson-by-lesson com- 
parisons (Dept of the Army, 1975, p. C-4), but an independent 
examination "of the data and analysis procedures revealed 
difficulties that njake some of the results equivocal (Avner 
and Misselt, 1977). Avner and Misselt do conclude, however, 
that the Army project's conclusions with regard to educa- 
tional effectiveness of CBE are generally supported by the 

''^available data. Related reports by Himwich ( 1977b) and 

Call-Himwich ( 1977 ) discuss the implementation and managem'ent 
of the Army project and'give a critical appraisal of the CBE 
lessons that were developed. 

Overall Methodology 

Conditions , The description of the training conditions 
or "treatments" which were established for comparison pur- 
poses is factually correct although it may be too brief to 
give a clear understanding of the way in which they were 
implemented. It is important to emphasize that the ISD 
revision of the four Special PuV-pose Vehicle courses (i.e., 
the shreds) included not only the first four blocks common to 
all shreds 1} termed the common course segment) but also tne 
remaining blocks- of each shred. The revisions to the common 
, course segment were more extensive in that all lectures were 
replaced by PLATO lessons and other self-paced media, but the 
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. contents of the "shred areas" (i.e., the blocks of each 
'course following the common course segment) were also changed 
to reflect the revised course goals (established by the ISD 
team in their analysis of job requirements). The only use of 
PLATO in the shred areas was in the administration and scor- 
i^ng of block- eaams; no CBE lessons were used. 

The revision of the shred^ areas, was cdnducted concur- 
rently with that of the common course segment. The B shred 
revision was completed first (note that according to Figure' 2 
in Chapter II, the D shred had the fewest blocks) so opera- 
tional' trials of the revised version could begin 15 Jan 1975. 
The D shred revision was the next to be completed with the 
first trial class starting on 13 March 19^5. The C shred was 
ready for its first class by 22 May 1975,, while the A shred 
(the longest according to Figure 2) was not completely' 
revised until 13 June 1975. 

It was this staggered completion schedule that Refined 
the treatment conditions. Because the A sTired revision was 
the last to be completed, classes enteHing that course 
between January and June of 1975 were given the conventional 
version of^he^^xricjilum. These classes wfere the only ones 
assigned to the NP (conventional) condition. Classes enter- 
ing the C and D shreds between January 1975 and the time that 
the shred area revisions were complete were assigned to the 
CP condition. The PB condition was made up of the B sh^pd 
classes and those classes entering the other shreds after 
their respective revisions had been completed, the BL con- 
dition included classes from all four sh^ed-s which, of 
course, received conventional trainings during the common 
course segment as well as the shred areas. 

Because of the confounding between shred assignment and 
condition (e.g., UP consisted only of A sftred students)-, 
interpretation of the results rests on ' the assumption that 
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there were no consistent differences among the shreds in 
terms of student aptitude, teaching ability of instructors, 
and other factors likely to affect performance. The fact 
that the treatment conditions were defined in this way, 
rather than by random assignment or some other j form of con- 
trol, is a threat to the.internal validity of the design 
(Campbell & Stanley, 1963). However, it exemplifies the 
- kind of compromise between the requirements of research and 
those of operational and logistic concerns that, unfortun- 
ately, must be made when an operational orientation is para- 
mount. 

Student population . As indicated by the authors, the 
differing sample sizes were due entirely to the staggered 
implementation of the ISD revi-sions to the various shreds. 
This is one area in which the constraints placed on the 
evaluation by the operational orientation of the service test 
were particularly severe. If it had been possible for the 
evaluators to control the rate of implementation, more uni- 
form sample sizes could have been obtaine^d. 

In general, overall sample sizes are judged to be suf- 
ficient to provide statistical tests of sufficient power to 
deteci the major effects likely to be of interest in instruc- 
tional settings. Detailed power analyses are given for indi- 
vidual comparisons later in this critique. The probability 
of detecting true differences of at least one standard devi- 
ation between means for the major groups (PB,CP,NP, and BL) 
was at least .95 (for an alpha level of .05). The prrobabil- 
ity of detecting smaller differences likely to be^ of research 
interest is, however, quite small (no higher than .56 for any 
comparison involving the NP grouj in detecting a true dif- 
ference of at least .5 standard deviations, again with alpha 
of .05). 
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III. INSTRUCTIONAL EFFECTIVENESS 

When the agencies most closely involved with the service 
test were surveyed regarding their priorities for the evalu- 
ation, the question of instructional effectiveness was seen 
to be of prime importance. It is not an easy question to 
answer— given the constraints arising from the "operational" 
orientation of Phase II--but those persons responsible for 
the evaluation have demonstrated considerable resourcefulness 
in. dealing with the problem. Chapter III gives a well organ- 
ized and informative account of the various data collection 
procedures and analyses used in addressing the instructional 
effectiveness issue. 

Special Purpose Vehicle Course 

Specia l purpose vehicle course procedures . This section 
gives a concise description of the variables measured and ' 
the. ^procedures used in data collection. No further comment 
ii' nee^d . 

Block exam scores' . As mentioned in an earlier section 
of this critique, the block exams in the common course seg- 
ment' and the shred areas had to be revised to reflect scope 
and depth-of-content changes resulting from the ISD effort. 
Once the revised courses were fully implemented, all students 
received the same instruction during the four blocks of the 
common course segment and were tested using the revised block 
exams. During the period when the evaluation data were col- 
lected, however, only the students in the PB condition were 
given the revised ex ams--students in the NP and CP conditions 
took the original (unrevised) versions. It is for thi^ rea- 
son that the< PB condition could not be included in the analy- 
ses of variance summarised in Table 3. 
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A comment is in order regarding the use of ANOVA to 
perform separate univariate analyses for each of the four 
block exams. Unless the variables (e.g., the block exams) 
are independent of each other, it rs inadv^isable to conduct 
separate univariate hypotheses tests for each of several var- 
iables measured on the same set of subjects. Another pos- 
sible analysis in such cases is multivariate analysis of var- 
iance (MANOVA) (see Tatsuoka, 1971). If the comparisons had 
been independent and if group sizes had been roughly equal 
(with the same total sample size), the tests would have had 
•the power to detect a .moderate effect (one accounting for 
5.88% of population variance) with probability of at least 
.96. 

A final comment on the reporting of the block exam 
results is that no information regarding the reliabilities 
of these exams is provided. At a minimum, it would have been 
useful to have repori .d the number of items on^each exam 
along with the means and standard deviations given i • Table -jg 
2. With this additional information, it would be possible to 
estimate test reliability using the KR-21 estimator. 

Course completion time . The rate at which students com- 
plete the course is an important variable in assessing the 
efficiency of the ISD revisions, but it does not directly 
measure their effectiveness. Presumably it is an indirect 
measure, however, becapse students would not progress from 
block to block if their instructio'n were so ineffective as to 
p.^event them from passing the block exams. 

Time saviAgs due to the course revisions were computed 
by comparing mean completion times under the revised course 
with the baselin^ completion times under the oi'iginal course. 
Comparisons were madp within each shred for the common course 
segment (which included PLATO instruction) and for the shred 

taken as a whole (i.e., including the shred area blocks). 

\ 
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It is apparent from Table 4 in the ARHRL report that time was 
saved in each of the common course segment blocks, but that 
some shred area blocks tob1<"^long(^r to complete under the ISD 
revision than under the baseline version. This latter obser- 
vation reflects the ISD redistribution of course content in 
some of the shred area blocks—some blocks were shortened 
while others were lengthened. 

Table 5 in the report presents the mean hours saved per 
block and the percentage of the baseline completion time that 
was saved in each shred during the common course segment. It 
, is stated in the text of the report that 28.5% of the common 
course segment baseline time was saved in the ISD revision. 
It should be recalled from earlier discussion that some of 
this time savings is due to moving some content items to 
Block X (home study) and much of the remainder is due tc the 
introduction of group-pacing to replace lock-step pacing. 
It may also be that the use of PLATO contributed to this 
<^verall time savings by saving some time above and beyond 
t-/gt expected by delivering the same instruction with other 
self-paced media. Additional discussion of this point is 
,'iven in Chapter IV. 

Eli minations . It was reasonable to plan to examine the 
frequency of academic eliminations in the various treatment 
conditions, but, as stated, there were so few eliminations 
during the evaluation period that analysis was unnecessary. 
Washbacks . No further comment is needed. 
Absenteeism . The number of absences is another of the 
course parameters which is easily measured and which permits 
comparisons among the various conditions. Its relevance to 
the question of instructional effectiveness is not clear, 
however, as it seems to be more of a measure of motivation 
than a measure of learning. As such, it might more appropri- 
ately have been discussed in Chapter V along with the other 
Indices of instructional impact. 
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The absentee data were analyzed by the Chi square test 
for independence.. The total number of absences and the mean 
number of non-abseitGes were compared across conditions in a 
series of two-way classification tables. Unfortunately, the 
discussion of how the mean number of non-absences was com- 
puted is given in too little detail for us to evaluate it^s 
logic. Without additional information, we are unable to ver- 
ify the correctness of the procedure or the accuracy of the 
results . 

Regardless of the outcom es of t he analyses of absentee 
frequencies, the supplemental ANOVA on number of hours absent 
seems to have been conducted appropriately. No significant 
differences among the conditions were found on this measure"^ " 

Special individualized assistance .- While the extended 
median test is applicable in the case described, use of 
Scheff*'s multiple-comparisons approach would have permitted 
the £ost hoc conclusion (that the CP students were given more 
assistance) to have been made explicitly ratiier than as a 
qualitative observation. 

hield evaluation. The first observatio;i likely to be 
■ made in a careful reading of this section is that there is a 
discrepancy between the number of NP students purportedly 
studied in the field evaluation (28) and the sample size 
reported for earlier analyses (21). It is our understanding 
that several students who were trained under the NP condition 
vere omitted from earlier analyses because of incomplete 
data. Missing data oji other variables should have no bearing 
on field evaluation analyses, however, so it seems appropri- 
ate that these students have been included in the NP sample. 

The discussion of the analyses of the field evaluation 
ratings data is fairly complete, and the analyses seem appro- 
priate for the available data. The major finding was that 
there were no significant differences auong the conditions 
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with regard to the overall performance ratings and a derived 
score based on the ratings for the 33 individual target 
tasks. A supplemental analysis showed that supervisor rat- 
ings on four of these tasks were significantly related to the 
overall performance rating for those students whose duty 
assignments were at the apprentice level. 

Discussion of course effectiveness . The discussion in 
this section addresses the question of whether or not the. two 
PLATO-using treatment conditions (PB and CP) were instruc- 
tionally effective. The premise on which the arguments seem 
to be based is that the PB and CP conditions should be con- 
sidered to be "effective" if they fare as well as the MP and 
BL conditions oh the various indices of course effectiveness 
that were reported. Accepting this premise assun^es that such 
measures as number of washbacks, rate of absenteeism, hours 
of SIA time, and instructor ratings in the fi^ld are adequate 
indicators of effectiveness. While most of these measures 
are valid for the desired purpose, many of them are insensi- 
tive (in comparison to tests) and are of limited value in 
discriminating among treatment conditions. Because the PB 
comm'on course segment block exams were different from those 
taken by the other groups, they could not be used for compar- 
ative purposes. Fortunately, the CP group took the same 
block exams as did the NP and BL groups so that a more .sensi- 
tive measure of CF condition effectiveness was available. 

Inferring from the pattern of results on the set of 
relatively insensitive indicators described above, the 
authors concludeci that the PB condition was at least as 
effective as the NP and BL conditions. Likewise, they con- 
cluded that the CP condition was "effective" — according to 
the criterion described above--because it fared as well on 
the various measures (including the block exams) as did the 
NP and BL conditions. 
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Although the discussion in this section of the report 
appears to us to be accurate, it suffers from lack of clar- 
ity. Furthermore, the- arguments do not se^m co^nyincing . We 
believe that much of this problem stems from the fact that 
the authors chose not to present the results found with 
another measure--the special topical test. Because we feel 
it provides additional insight into the question of course 
effectiveness, we wish to supplement the AFHRL report by dis- 
cussing this test and the results obtained with its use. We 
have had access to this information due to our role as data 
analysts in assisting with the service 'test. 

A special topical test was constructed by the service 
test evaluation team for the purpose of attempting to isolate 
the effects of the PLATO system from those of the othe^ 
changes made during the ISD redesign of the curriculum. This 
topical test — mentioned 'briefly in the report under the head- 
ing "PLATO System Ef fee t iveness"— consisted of 30 items- test- 
ing, material covered in the CBE lessonsand 20 items over 
topics presented by other media. The original intent of the 
evaluators was to compare the various treatment groups on the 
basis of scores for each of these 1;wo sets of items (i.e., a 
PLATO item score and a non-PLATO *rtem score).' Presumably, it 
would be possible to determine from these comparisons whether 
PLATO had effects above and beyond those of the other "ISD 
influences". Because of excessive item difficulty and other" 
technical problems, however, the PLATO item and non-PLATO • 
item sub-scores were highly unreliable (coefficient alpha 
reliabilities of .55 and .38, respectively) and these results 
were not reported (see section on "PLATO System 
Effectiveness" ) . 

While it is appropriate that the sub-score results were 
omitted from the report, a second set of analyses based on 
the somewhat more reliable total scores (i.e., the sum of the 
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PLATO and non-PLATO item sub~scores) gives valuable informa- 
tion on the question of course effectiveness, the question 
addressed in this section. The reliability of the total 
scores falls short of what would be desirable for most pur- 
poses (coefficient alpha = .65), but. If interprc ;ed cau- 
tiously, the results can be used as a check on findings 
obtained with other measures. 

The special topical test was actually administered three 
times during the flow of the course— once as a pretest, once 
at the conclusion of the common course segment (posttest 1), 
and once at the end of instruction in the shred areas (post- 
test 2). It had been planned by the evaluation team to com- 
pare the treatment groups at each of these three point's in 
time, but for the present purpose it is sufficient to con- 
sider only the pretest and posttest 1. 

There were no significant drfferencc»s among, the groups 
on the pretest (F(2, 1i»5)= .434, p=.649). an indication that 
they were homogeneous with i ^gard to initial knowledge of the 
topics covered. As can be seen from Table 1 (in this cri- 
tique), the MP group had the smallest mean performance on 
posttest 1. , 



Table 1 

Group Means and Standard Deviations 
on Posttest 1 Total Scores 



ijroup 


Mean 


Standa. d 
Deviat ion 


n 


FB 


28.111 


4.696 


81 


CP 


28.727 


5.398 


44 


NP' 


23.565 


4.998- 


23 
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A one-way fixed-effects ANOVA on posttest 1 total scores 
is summarized in Table 2. The effect underlying the signifi- 
cant F statistic accounts for 11.2? of total sample variance. 
The power of this test to detect an effect of moderate size 
(accounting for 5.88? of population "v^r iance) is only about 
.76, so a smaller effect might easily have missed detection. 
Post hoc analyses using the Scheff* method show the PB group, 
performance to be significantly higher than that of the HP 
group (p = .001).- The CP group also had significantly higher 
scores than the NP students (p < .0005), but the difference 
between the PD and CP groups was not significant (p = .803) 
and is most likely due to chance effects. 







Table 2 




Summsry of 


ANOVA 


on Posttest 


1 Total Scores . 


Source 


df 


MS 


P 


trtmnt 


2 


225.741 


9.177 .0002 


error 


145 1 


•24.598 




V 









The special topical test results seem to indicate that 
the ISD-revised course taken by the PB students was more 
effective' than the original instruction on these topics. The 
fact that the CP group also fared well relative to the NP 
group is due to the combined effects of the CBE materials and 
supplementary discussions/review provided by the instructors. 
It is interesting that this supplemental instruction did not ' 
raise the scores of the CP students significantly above those 
of the PB group even thoiJgh more training time was expended 
(see Ta^le 4 in the report). It should be remembered that 
coefficient alpha for the special topical test was only .65, 
but, on the other hand, the major effect of unreliability in 
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ANOVA is to obscure the differences among treatments by 
reducing statistical power (for a discussion of this point, 
see Cochran, 1968; Misselt, 977). It is important that sig- 
nificant differences were faunti in spite of such unreliabil- 
ity. Although there may be other confound ing ' factors (e.gT^' 
differing levels of experience/effectiveness among intructors 
for the various conditions) which Qould account for these 
results, they seem. to add direct support for ''the conclusion 
drawn in the report itself that the PLATO-using conditions 
were "at least" as instr uc tionally effective. as the non- 
P'.ATO versions. 

In addition to the authors' decision to .base their dis- 
cussion of course effectiveness on what we consider to be 

ft 

relatively insensitive measures, another proDlem detracting 
from the clarity of the presentation is their tendency to 
intersperse discussion of "efficiency" issues within the 
context of the "effectiveness" question. It is desirable to 
compare the efficiency of the various treatment conditions 
but it is important not to confuse measures of efficiency 
•(e.g., the completion time required in the various blocks) 
with those of effectiveness. With regard to instruct irnal 
efficiency, there is ample evidence that the revisions did 
reduce training time. This, after all, was the major goal of 
the ISD team. They did not set out to teach more content or 
tea^h to a higher degree of mastery. Instead, they attempted 
to make the common course segment instruction more efficient. 
They apparently were successful in this effort. 

PLATO System Effectiveness 

Procedures . The evaluation team attempted two proce- 
dures for assessing the effectiveness of the PLATO system (as 
used at Chanute) apart from the effects of other important 
variables in the environment. The first of- these, mentioned 
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only briefly in the report, was a plan to use the sub-scor^.\ 
on the special topical test to look for differences in learn- • 
ing between the treatment groups on two sets of topic s--those' 
covered by PLATO lessons and those taught using more copven-- 
tional media. Any differences among the groups on the non- 
PLATO item sub-scores presumably were to' be taken as evidence 
..of differential effectiveness betwe3n the revisions and the 
original instruction on these topics. If, for example, tiie 
PB group were to do no better than the NP group on the non- 
PLATO item subtest, it Would have been an indipation that the 
ISD procedural changes inherent in the PB condition had no 
special effect. If, meanwhile, the PB-group had done signi- 
ficantly better than the NP group on the PLATO item subtest, 
it would have been taken as evidence that Chanute's use of 
PLATO had effects above and beyond those of the remaining ISD 
influences. This plan was not implemented because of the low 
reliability of. the sub-scores (see discussion in the previous 
section of this critique). However, even if the tests had 
been more carefully developed so as to have adequate relia- * 
bility, it is noV immediately clear that' it viould have been 
possible to separate the effects of . PLATO from those of the 
other environmental changes using the plan outlined above. 

The second procedure used in attempting to determine the 
effects of the PLATO system (i.e., while controlling other 
variables) was a special comparison betw^^en a group studying 
four of the^ PLATO lessons and a group given four programmed 
texts over the same set o"^ topics. Because the programmed 
texts used electrostatic copies of screen displays (or 
frames) from the PLATO lessons, the extraneous effects 'of 
subject matter and lesson strategy were controii^ci (Salomon & 
Clark, 1977). The comparisons were based dn end-of-lesson 
test scores and the time required' to complete the lessons. 
The students studying the programmed text versions were also 

\ . ■ 
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asked to state their preference between programmed texts and 
PLATO (they had used PLATO to study other topics). 
The discussion of th-ese procedures is clear and concise, but 
it would have been useful to add some desicr,ipt ion of the^ 
data collection procedures and some assessmeQ%^r'^t^he_^(, 
reliability of the measures used (including the completion 
time measurements). . ' ' . r ^ 

Results . No significant differences' in lesson test per- 
formance were found be'tween the PLATO" and Programmed Text 
(PT) comparison groups*. jOr^ concern "to be noted is that the 
tests* used wer© originally designed to be criterion refer- 
^nced tests. Because criterion referenced tests tend to give 
a "ceiling" effect and have low variances, they are generally 
not very suitable for compar ison purposes (Popham, 1971). 
^ Even so, a more senstitive measure could have^ been<rObtained 
by summing the item scores on the four^ lessorj tests to give 
a single overall score. 

In the absence of information about the reliability 
of the completion .time data, little can be said about the 
results obtained. It is important, of course, that the 
PLATO group had fewer individual failures and that the PT 
group indicated a 'preference to studying lessons on PLATO. 

Discussion of comparison findings . The discussion in 
this section is carefully reasoned and accurate. The conclu- 
sion that the PLATO lessons were more instruct ionally effec- 
tive than their PT counterparts aeems Justified by the data 
I. presented 4 The caution about oyergeneral izat ion of these 
'results is an appropriate one, howe'yer. 

PLATO Cour.seware s 
. I 

Procedures ^ jAs * ind icated' by the report authors, there 
were insufficient resources to develop parrillel non-PLATO 
instruction on eack of the topics targeted for/CBE. Since no 
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Other /^lesson materials covering the. appropriate content were 
available, it was impossible to do a comparative evaluation 
on a les?on-by-lesson basis* The alternative was to deter- 
mine the extent to -which each lesson was capable of bringing 
students to a pre-determined level of performance. 

Once a lesson was 'shown to have been effective instruc- 
, tion for a suitable va]^idation sample, it was said to be 
"validated". The proportion of students passing each les- 
son's Master Validation Exam (MVE) was monitored continuously 
during the evaluation of Phase II. When a validation attempt 
failed (i.e., more than 10% o^f a 2^30 student sample failed 
to^^ass the MVE), th>e lesson was revised in the hope of mak- 
ing it more effec^tive^ Once a validation attempt was suc- 
cessful, no mofvj revisions were made (this procedure was 
based on the premise that lessons should be deye'! ')oed as 
•Jlean" as possible initially, then making only tho^e addi- 
tions which are necessary for them to validate). 

A check on the "appropriateness" of the decision> to con- 
sider a lesson as having validated was made by monitoring 
"post validation" failure rates-. If the failure rate was too 
high (greater thari 20%), the lesson ^as to be revised and' ^ 
subjected to new valida.cion trials. Failure »"ates between 10 
and 20% were an indication that ^ lesson should be monitored 
carefully. Only if no more than 10% of the post validation 
studertts failed the MVE was the J.esson considered to be 
instructional ly stable. 

Results . The .fact that four lessons definitely needed 
revision and eight others may ' e needed to be reworked 
indicates, that the original va. Jation criter ior^was too 
lenien;K That a systematic validation procedure was followed 
at a/l is certainly commendable, but the procedure used seems 
arbitrary and intuitively based. A more scientific valida- 
t^^or^ procedure has been developed by Tatsuoka ( 1 977 ). It 
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applies a Bayesian statistical model to the data from a vali- 
dation attempt and derives an estimate of the probability 
that the lesson will "remain" validated in the population 
from which the initial sample was drawn. " It was possible 
with this procedure to predict which lessons would be most 
likely to "de-validate" over time. The data given in the 
AFHRL report confirm these predictions. A comparison of 
Chanute's validation approach and the proposed alternative is 
given in the report by Tatsuoka (1977). Tatsuoka's report 
also discusses the shortcomings of defining "validation" in 
terms of test scores; a poorly-constructed test may not be an 
adequate instrument for judging the effectiveness of a 
lesson. There^ is a need for establishing a better validation 
criterion. - 

The analysis of on-line time data is probably more rele- 
vant to the topic of instructional efficiency than it is to 
instructional effectiveness. In any event, however, the 
statement that further time savings could be realized from a 
self-paced course structure is; quite justified. 

Discussion of lesson performance results . The discus- 
sion in this section is well-stated and complete. No further 
comments are needed. 

/ 

Instructional -Effectiveness Conclusions 

^The major conclusion regarding effectiveness is that 
the PLATO lessons and other materials and procedures intro- 
duced in the new course led to adequate student learn ing . We 
believe the available data supports this conclusion. 

A second conclusion is that the revised course is more 
efficient because it requires less training time to complete. 
Part of this time savings may be due to such things as depth- 
of-instruction changes, moving some cf the material from the 
common course segment to the shred areas or to Block X (i.e., 
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home study), and the change to a group-paced format. It is 
not possible, therefore, to attribute all of the time saving 
to more efficient teaching strategies or the standardization 
of instruction, but it is llOcely that these account for some 
of the efficiency advantages. 
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IV. COST FACTORS 

This chapter demonstrates the difficulties of determin- 
ing true operating costs in realistic operating environments. 
Actual instructional settings are rarely composed of indepen- 
dent accounting units. Thus iJL_is usually necessary to infer 
costs as some fraction of a total^^^ich includes the expenses 
of unrelated activitie.s. Experimental operations also are 
rarely able to make use of the savings that would accrue from 
saturation-use of delivery faciliti4s. . Under experimental 
conditions, there is a need to provided-sufficient unscheduled 
resources to permit exploitation of'research opportunities 
which are discovered during a project. 'In a "production" 
setting, all available resources would M fully utilized with 
the result that^costs would be shared by more users, and 
cost-per-unit-instruction, would be decreased. 

Instead of following the headings and format of Chapter 
IV, in which all costs are separately categorized, this sec- 
tion of the critique converts presentation costs into a sin- 
gle form for ease in gaining an overview. 

Using industry standards, costs may be determined on an 
annual basis by assuming electronics hardware„,to have a seven 
year life span, classroom improvements to have a ten year 
life span, and interest costs to the ^Government to be 7%. We 
then find the total capitgl expenditures of $191,688 have an- 
imputed annual cost of $35,568 per year, the $4,803 spent. for 
facilities preparation have an annual cost of $684, and oper- 
ational costs are $89,933, for a total annual cost of about 
$126,185. These figures may also be interpreted as costs per 
terminal of $4,206 per year (with 30 terminals) or as costs 
per terminal hour of $3.85 (using the figure of 1092 hour's 
usageper terminal per year cited in Table 20 of the report). 
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Since less than 1% of the annual* cost of terminal oper- 
ation is a function of operating time (e.g., the cost of 
electricity for operating the terminal), the cost per hour of 
a terminal could be lowered substantially by higher utiliza- 
tion rates. The average weekly use of the terminals was less 
than 25 hours, a figure that could liave been doubled with 
moderate ease in a full-scale training situation, "ihe hourly 
terminal ,cost is of interest because it is this , figure which 
determines whether a savings in training time from use of CBE 
is sufficient to make use of CBE cost-effective. 

Based on the estimated student time savings of 1 1/3 
training days (or $64) cited in the section on ."real sav- 
ings", time savings would pay for only about 16.6 hours of 
CBE time when terminals were used for no more th..n 25 hours 
per we^k. If terminal usage were doubled, as suggested b^ 
the-^ AFHRL report, the hourly rate per terminal would decrease 
and the observed time savings would have paid for over 33 
hours of terminal time. Since these times bracket the 
observed s.tudent on-line times in the course, it appears that 
CBE was close to being as cost-effective in this situation as 
programmed texts. A complete analysis of such a comparison 
wouia, of course, be possible only If data for larger-scale 
implementations of both approaches were available for this 
situation. Also needed would be more detail on indirect 
costs associated with use of CBE and alternative media at 
Ch^anute. Unfortunately, the basis for many of the cost fig- 
ures Yeported in this chapter is not given in sufficient 
detail to go beyond the analysis given by the authors. Given 
such added detail, potential savings in instructor time and 
marginal benefits in increased student-instructor satisfac- 
' tion could al:^.^be . evaluated . It is possible that the dis- 
tributed instructor time-savings described by the authors 
could be applied to a heavier student load in some situa- 
' tions, for example. 
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In summary, if the ApyRL estimate of a 10* time savings 
due to CBE is correct, it appears that the experimental 
implementation was close to being cost-effective and that the 
more complete resource utilization possible in a full-scale 
implementation might be cost-effective even with currently 
available hardware. 
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V. INSTRUCTIONAL IMPACT . 

Chapter V gives an extens.ive and careful discussion of 
several issues relating to the incorporability of PLATO in 
the Air Force technical training environment. It is a very 
informative and well-organized apcount. However, for the 
-sake of convenience, our discussion uses a system of headings 
and sub-headings which is slightly less detailed than that 
used in the chapter. 

Introduction 

^ The rationale for including the study of student and 
instructor attitudes in the evaluation is'quite valid. The 
attitudes of these two key groups are impprtant indicators of 
the overall acceptance of an innovation such as CBE; student 
and instructor acceptance are essential . for effective imple- 
mentation. 

Student Attitudes 
\ Shorty form survey . The similarities and differences 

between the PLATO and non-PLATO versions of the short form 
survey and the scoring procedures for both are appropriately 
described . 

Method, the procedures for data collection are 
described concisely. No further comment is needed. 

Results . Given the decision to combine the individual 
attitud items to form a single scale, the combinations chosen 
by the authors for the PLATO version and the NP version seem 
appropriate. The reliabilities of the resulting scales are 
sufficiently high, however, to suggest that the individual 
item reliabilities may also be fairly large. If so, combin- 
ing the item sco'-es into a single scale score may effectively 
"throw away" some information available in the individual 
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items. Tatsuoka, l^isselt, and Maritz (1977) report a number 
analyses of the short form survey response data which treat . 
each item as a separate variable. In a similar vein, the 
decision by the report authors to base their analyses on a 
random selection of 20 subjects from each condition does not 
make full use of the available data. Rather than "throw 
away" data from the PB and CP conditions, the authors might 
have used the unweighted means analysis or the least squares 
solution (Kirk, 1968, pp. 276-282). 

The first analysis reported was a comparison of the so- 
called "PLATO-using" conditions (PB and CP). It used a scale 
based on a combination of all five short form items and 
showed a significant drop in attitude as the students -pro- 
gressed through the blocks. The CP group held slightly more 
favorable attitudes than the PB group, but the difference was 
not significant. Both groups* attitudes remained positive 
throughout the course even though they were less positive at 
the end than they had been at the beginning. 

The second analysis also used only 20 subjects per 
group, but it allowed comparisons, among all three conditions 
on the scale derived from the three media acceptance items. 
As before, there were no significant differences among the 
conditions, although the use of all available data would pro- 
bably have increased the power of the test, for this effect 
and significant differences might have been found. 

The summary of open-ended comments is informative 
and is consistent with our understanding of both the student 
population and the CBE lessons developed at Chanute. 

Discussion . The hypotheses suggested in this 
section regarding the decline of attitudes over time are 
reasonable. The authors' hypothesis that student attitudes, 
may have been negatively influenced by technical packaging 
errors in the CBE lessons (e.g., typing, and spelling 
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errors, nonexistence or inability to access "advertised" 
branching sequences, execution errors, etc.) is consistent 
with results found by R.^^A. Avner (personal communication) 
in another context. .Avner correlated "quality scale" ratings 
of 23 lessons with student ratings of "helpfulness" for learn 
ing the presented material. 'The qual ity~ scale included 
references to technical errors so severe that^^they prevented 
students from completing a lesson, as well as minor 
.typographical errors that were only cosmetically harmful . 
He found a significant relationship between t^e two sets of 
ratings (r(21)=.923 with .95 Confidence range of .825 to 
.967). The Chanute materials contained a number of these 
technical problems during the data collection period. 

It is also plausible that the . PLATO-using students were 
subject to the "Hawthorne effect" during this study. This 
effect could have , caused an initial inflation in their 
attitudes toward CEE followed by a gradual decrease through- 
out the course. In any event, the attitudes toward PLATO 
were more consistently positive than attitudes toward 
instructors. This consistency may be a real advantage for 
CBE — especially during periods of mobilization when experir 
ienced instructors may be difficult to find. 

Long form survey . The description of the survey ques- 
tionnaire is quite adequate for communicating its essential 
features . 

Method. The factor analysis of the combined PB and CP 
survey responses is not reported in as much detail as might 
b^* useful for independent interpretation. Some additional 
detail is provided in this critique. 

The responses of the 200 PLATO-using students on the 
bo-item survey were subjected to a principal components 
analysis and the resulting components were rotated by the 
varimax method. The five factors that were extracted 
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accojunted for the following percentages of total variation: 
30.52, 6.05, 3-95, 3-63, and 3-21. Together they accounted 
for 47*09 percent of the total sample variation. 

The interpretations of the various., factors suggested 
by the authors may be over-simplified in that the factor 
labels apply properly only to the aggregate of items whose 
factor loadings are reported in Appendix D; they do not 
apply so readily when the loadings for the rema^ining items 
are examined. The "purity'* of interpretation implied by the 
various factor labels is attained through the expedient of 
ignoring those items which appear to be in conflict with the 
chosen interpretation label. To be sure, the items on each 
factor which are ignored are those which havQ "low" loadings. 
The definition of "low", ho.wever, differs from one factor to 
another; the items the authors choose to ignore on ^ome 
factors actually have higher loadings than those of items 
which were retained on other factors. 

Resul ts . The report authors computed "raw" scale 
sco'res on the various factors for each individual .by summing 
scores on those items whose factor loadings are reported 
in Appendix D. Since the number of items selected for 
inclusion differs from factor to factor (because of the 
differing definitions of a cutoff for loading values 
described- above) , the raw sums for each factor were divided 
by the number of items to give scores representinp, a person's 
"average" response. 

The interpretation of the mean scores for each factor 
scale (i.e., the discussion of the results in Tnble ?4) seems 
reasonable, but it is difficult to do an independent 
interpretation without also beinf, told the scales' standnrd 
deviations . 

Two sets of comparisons were completed usin>', the menn 
scale scores described above. The first of those was n 
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scale-by-scale comparison of the two PLATO-using groups whose 
data were the basis for the factor analysis. No significant 
differences were found using a series of independent groups 
t-tests for the five scales. Given the non-orthogonal nature 
of the variables, it may have been more appropriate to do a 
multivariate analysis (e.g., Hotellinrg's T 2 or MA^A 
followed by discriminant analysis) rather than a series of 
univariate t-tests (see Tatsuoka, 1971). 

The second set of analyses was a series of one-way 
univariate ANOVAs comparing the mean scale scores of the 
Conventional PLATO (CP), Early PLATO-Based (EPB), and ^ 
Later PLATO-Based (LPB) students. The distinction between 
the EPB and tPB students is that the EPB students Studied 
the ISD version of the common course segment before May, 1975, 
and the LPB classes studied the same materials (except for 
possible revisions) after May 1975. As was true of the 
earlier analyses, it would have- been preferable to have 
applied a multivariate technique" ( e .g . , MANOVA) rather than 
to have done a series .of univariate tests with non-orthogonal 
variables. If an overall multivariate analysis were to give 
significant results, the exact nature of the group differ- 
ences could be determined using discriminant analysis. 
Because of possible interrelatedness of the scale variables, 
the significant univariate ANOVA result reported by the 
authors might be an artifact. Even , taking this result at 
face value, it is impossible to interpret without a knowledge 
of the group standard deviations on the scale for factor 1. 
It is unfortunate that this information was hot reported. • 

Another analysis of the long form survey data focusea on 
the preferences of the CP and PB groups (the latter including 
both EPB and LPB classes) for each of the various media used 
in the course. Mean responses for these groups were compared 
on an item-uy-item basis using t-tests. The items in 
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question were items III-l to III-6 in the long form survey 
(see Appendix C), The earlier comments regarding use of 
multivariate techniques apply to this set of analyses as 
well. This caution noted, th^ only comparisons yielding 
significant t statistics were those regarding item XII-2 (PB 
students viewed ^PLATO lessons more favorably than CP students) 
and item III-5 (CP- students were even more favorably disposed 
toward laboratory instruction than the PB students). 

The graphical portrayal of mean responses tdVard the 
•various media showr in Figure 5 of the AFHRL report is a 
concise format for conveying the media preferences of the 
two groups. It would also have 'been helpful to have 
indicated the, within-group variation in stated preference. 
At any rate, one of the most striking observations*to be 
made from Fi^j^ 5 is that the average vehicle maintenance 
student has a strong preference for laboratory activities; 
he woul'd much prefer to -work with his hands than^to engage 
in learning activities^ that are more cognitive in nature. 

Items III-7 to III-18 of the long form survey (see 
Appendix C of the AFHRL report) asked the PLATO-using 
students to ind icate the ir emotional responses 'toward the 
PLATO system as used in their course. The individual 
t.-tests that proved significant and their interpretation are 
aptly discussed by the authors. Perhaps more interesting 
than these PB vs. CP comparisons is the graphic 
representation of the mean response patterns for the two 
groups shown in Figure 6. It is apparent that the overall 
response toward PLATO was positive. Information on. the 
group standard deviations for each item would have been 
helpful in determining the extent to which the reported 
means are typical of the individual student responses. 

The correlational analyses ind icating the positive 
relationship between attitude and perceived reliability of 
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the system are quite interesting and are consistent with 
our experience with other PLATO users. It is also 
interesting to note, however, that the attitudes of tnese 
students (regarding perceived reliability as well as other 
aspects of their experience with PLATO) were not related 
to their performance on block exams or the special topical 
test (see Tatsuoka, Misselt 4 Maritz, 1977). Perhaps one 
reason interruptions to service did not affect performance 
is that student learning was controlled by the mastery learn 
ing ct^iterion. If a student happened to earn a low score on 
an end-of-lesson test because of system unreliability 
during scudy of that lesson, he was routed back to the 
lesson's starting point. The mastery criterion served as 
insurance against ill effects of unreliability as well as 
those of other factors which may hinder performance. Hence, 
scores on clock tests were not affected by-system "crashes". 
It is not known whether interruptions to service may detract 
from perfDrmance in non-mastery settings. 

Discussion. The commentary given by the authors on the 
attitude results is generally quite reasonable. The 
conclusion that the automotive maintenance students were 
very accepting of PLATO as an instructional medium seems to 
be well justified. 

Tec.nical training survey . This section's discussion 
of the purpose of the technical training survey and its 12 
scales is adequate for an understanding of its application 
in- the service test More complete information on its' 
development and the results of its use in the broader con- 
text of Air Force technr.cal training presumably is forthcom- 
ing in_^a separate publication. 

Me thod . No .comment is needed. • 

Results . Sufficient usable data were available to 
perform 4x2 (condition x administration) univariate 
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repeated measures ANOVAs on eight of the 12 sets of scale 
scores. Although a multivariate analysis technique might be 
preferred, the authors' summary of the univariate results is 
clear and concise. Tc be more complete, the summary might 
have included tables of means and standard deviations for 
each administration to each of the four conditions studied 
(i.e. , BL, NP, PB, and CP) . 

The comparisons of tne automotive maintenance students' 
attitlides~'with-^hos'e of- o-ther_,i^^^^ training students at 

Ch^nute^ and within Air Training Command as a whole were 
conducted by computing confidence intervals around the 
observe^ mean on each scale for the students in each 
condition. Estimates of the Chanute and ATC population 
standard de/iations were used in computing the various 
confidence intervals. If the total sample mean for the 
Chanfute trainees (n=24840 or for all ATC trainees (nr12666) 
fell outside a particular .confidence interval, it was taken 
as evidence that the sample differed significantly from 
the respective population on the scale in question. Since 
the estimates of the population standard deviations are not 
reported, we are unable to check the calculation of the 
various confidence intervals. However, the procedure seems 
to be a reasonable approach to the analysis of these data. 

It might be wished that the authors had^ taken greater, 
care.;in summarizing the results in terms of completeness and 
accuracy, but they Nhave reported the data-in sufficient 
detail (see Appendixes F and O'that interested readers can 
search for other relationshi'p^ . 

' Discussion . No further comment is needed. 

Instruct or Attitudes 

Method . The description of the instructor attitude 

'J ' ' ' 

scale is adequate a^ background for presenting the results. 
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It would have beerl^helpful to have 'given a fuller descrip- 
tion of the sample of instructors since it is apparent that 
they differed with respect to the duration of their exper- • 
ience with. the PLATO-based training system. 

Hesults . It is commendable that reliability estimates 
were computed for those scales where it was possible to do 
so. Unfortunately, however, there are some difficulties with 
the statistical techniques used on these data. For example j 
the current description of Table 28 appears to be somewhat 
misleading, as it describes "t-tests for the attitudinal data 
-analyzed- -as -.re paatjBd jnea^uj^^ with correlated observations." 
This would seem to imply that a series of correlated^~t.-resYs"^ 
had been computed on- the scaled responses for the two admin- 
istrations of the survey. However,* the analyses actually 
reported in 'Table 28 were, independent groups ^-tests. Since 
there is an overlap ot 10 instructors who were surveyed in 
both administrations, *it is likely that the assumption of 
indepenaence is violated in these data. Hence, thfe validity 
of the probability statements in Table 28 is n;uestidnable . A 
more defensible procedure for ^assessing attitudinal changes 
would be to concentrate on the responses of those instructors 
*who had been surveyed twice. A multivariate "analogue of the 
correlated t-test could then be done on the two sets of scale 
scores. ^ 

Responses to the items underlying the 'scale labeled 
"PLATO affect" (i.e., items 23 to 34 in Appendix H) were 
examined more closely in supplementar analyses. Wtiile a 
more complete description of the and categories, 
in the C^i square analysis would be helpful, the Chi square 
results confirm the earlier jt-test 'results regarding the 
"PLATO affect" scale. Figure 9 gives a graphic portrayal ' 
of the fact that attitudes became more negative at the time 
of th3 second administration whereas they previously had . 
been generally positive. , . ^ ^ 
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Discussion ^ The authors' in'ter pretation of the declin- 
ing attitudes observed in the survey responses seems 
plausible; a decline in perceived role importance could 
result in less favorable attitudes toward CBE. The authors' 
suggestions regarding increased instructor involvement 
appear to be well founded and should be investigated empiri- 
cally in future CBE applications. ^ 

iSome additional comments seem in order regarding the 
interpretation of these_r^esults. First of all, the timing of 
the questionnaire administration should be noted. The 
initial administration came in February 1975 after only a few" 
week^ ~o~f ~ope rati-onai- us e-af-the-P-LAT-0- -syst em_in_JtKe__r_ejy is ed _ 
common course segment, and the second came in July of that 
year. Although it occurred several months into the , 
operational trials, the ' second administration must still be 
considered to have been quite early in the life span of the 
Phase II operation. Many instructional and coding problems 
remained in the lessons (most of which had not yet reached 
Chanute's validation criterion) and the group-pacing of the 
shred areas had only recently been completed. In short, 
there were still a number of "bugs" in .the newly-rrev ised 
courses which may have negatively influenced the' instructors' 
attitudes toward' PLATO in the time frame of the second survey 

Another item suggesting that the decline in instructor 
attitudes may have been due to tempprary factors is that 
more recent interviews with instructors (Klecka, 1977c) 
revealed a generally positive attitude .toward PLATO. It is 
not known how many of the individuals interviewed by Klecka 
had been part of the instructor force' at the time of the 
AFHRL survey. It is possible that substantial "turnover" 
had' taken place so that some newer instructors learned a 
"PLATO role" without having first become accustomed to a 
more traditional role. Without any previous role pattern or 
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expectations, new instructors may adapt more easily than did 
the original staff* Perhaps, however, the most direct way 
of testing the hypothesis that tlie attitude decline, observed 
in the AFHRL survey of July 1975 was due to temporary 
factors would be to administer that survey a third time 
(i.e., to the current group of instructors). 

Impact on Training Activities 

Procedure . It is highly commendable that observational 
data were collected and reported by the service test evalua- 
tion team, but some aspects of the method of gathering these 
data are not clear from the report's description. In partic- 
JUJar_,_Jt _can_not be determined from the report whether the 
observation categories were used in making the obselrvations ~ 
or whether they were arrived at after the data had been 
collected. Although a highly structured observation instru- 
ment would have been impractical, it. would be r*eassuring to 
know that the observers had at least been instructed to 
attend to some general categories of beh^'avior and that there 
had been some ground rules for the classification and 
reporting of observations to minimize* observer bias. 

Results and Discussion 

Conduct of training ; . The description of the adjustments 
to the standard operating procedures (made necessary by 
Chanute's CBE implementation) ,is enlightening. ' The informal 
"sign-up sheet" scTieduling system seemed to work fairly well; 
however, it is not known whether it would continue to be a 
smooth and convenient way to allocate terminal time 'under 
conditions of greater demand for this resource. A more 
sophisticated scheme may be necessary in a full-scale 
operational environment. 
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The report indicates that the Chanute site occasionally 
suffered from shortages of computer memory space (known as 
extended core storage or ECS). It should be noted that such 
problems occurred only when ECS usage at Chanute exceeded 
the allocation guaranteed to them. The two general ways 
that this problem cafi be solved are (a) making more ECS 
available for each user (by adding expensive hardware to the 
system's mainframe or by restricting the number of- users on 
the system), and (b) reducing the amount of ECS required by 
each user. The AFHRL report implies that if PLATO were an 

• Air Force resource, memory problems would be eliminated. In 
order for this to be a solution, restrictions on the number 
of users on an Air Force PLATO system could allow each user 

, to---occupy--mare~memory--^ptiohably_ wi,t costs per 

user. The second approach to the problem would require the 
average Chanute lesson to be reduced to one-half or 
one-third of its current size. "The latter solution would 
consume slightly more computer disk storage, (a relatively 
- ; inexpensive resource) but' would require a major modification 
^ ' effort to partition existi-ng lessons into a greater^ number 
of smaller modules. Future ECS problems could be avoided 
by careful planning early in the course of a project. The ' 
most viable approach would be to keep the ECS requirements of 
discrete lesson segments under 2000 words*./ 

Student interaction . The description of student inter- 
action in the traditional and CBE classrooms is useful. It 

' affords a much fuller picture, of the transactions in the 
training environment than could be had through test scores 
and attitude responses alone. 

The classroom interaction rate data for the vehicle' 
training lecture sessions is especially, informative. Based 
on data supplied by the AFHRL report, the following comparison 

. can be made between the number of questions answered by 
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students in a PLATO class and in a lecture class. According 
to this section, 11 students answere^d 77 questions in four 
hours, i.e., less than two questions per sti.dent per hour, 
summing the information in Tables 17' and 31, one can 
calculate that PLATO students answered 155 questions (even 
more when reviewing is counted) in an average of 369 
minutes, or more than 25 questions per student per hour. 

We take issue with some of the comments made by the 
report authors in comparing CBE and conventional instruction 
For example, their criticism that in comparison to instruc- 
tion by lecture, CBE lessons are less easily adapted to the 
needs of students may be true^ but the arguments byV which 
they draw this conclusion are .not valid. "Adaptability" as 
used in Chapter V seems to mean the Ability to reshape the 
delivery of mate£_ial to'^a class.* This is rarely attempted 
in CBE "because" of the availability of " ind ividuairzatioh" , 
me'aning adaptation for each^s^tudent rather than for each^ 
class. "Adaptability" also implies the ability to half 
normal training when a class-wide misunder stahd ing is ^foted 
and to then re-teach any necessary concepts. Because\the 
lecture format is-" not standard ized ,^ this • adapt ability 
fulfills a very great need — it compensates for omissions or 
inadequate explanations by some instructors. With well- 
tested standardized ^edia pre»sentations , however , this ^ 
"total adaptability" becomes^ unimportant . Furthermore, 
Chanute CBjE lessons were not, in our opinion, de^signed for 
either adaptability or individualization, so that the AFHRL 
report comments may be appropriate if applied to Chanute CBE 
lessons, but not necessarily to CBE in general. 

Because both^'the PLATO"" system and the now-f reed-(:rom- « 
lecture instructors can gather data about student qifestions 
as '-an aid for revision, we do not agree that, " [a] ttempting 
to keep a record of questions . . . would be a difficult, 
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if not impossible, task for a, single instructor." Revisions 
based on just this sort of data can give CBE lessons a huge 
advantage over other media. Furthermore, the capability of 
the PLATO system to collect, store, and organize data for 
formative evaluation or individualized training is certainly 
greater than that of the typical classroom lecturer. Never- 
theless, the instructor 'who monitors a CBE classroom is in a 
position to observe a number ,of problem areas in the CBE 
lessons and, if he has the inclination to do so, can supple- 
ment and clarify the student interaction data collected 
by the system itself. Perhaps the Chanute instructors could 
be given some additional training in techniques for gathering 
fo^rmative evaluation data;cthe recently-added "TERM-comments" 
feature, for example, cduld be used quite effectively by 
instructors who have observed areas of needed impJrovement in 
the CBE lessons. , . 

Instructional content . The observation that students in 
this' population were less - interested in theory or abstract^ 
principles of engine operaj^ion than they were in laboratory 
tasks confirms the result noted earlier from their responses 
on the long-form attitude survey. , We would also underscore 
the observation that the ability to repeat in a tireless, 
standardized * way drills and exercises which a human 
instructor would disdain and execute poorly is one of the 
most valuable assets of CBE. 

In structional styles . Th^ comments in this section with 
regard to the differ ing .teaching. -styles of instructors and 
the adjustment that students h?d to make as a result are 
quiti interesting. It would seem that this phenomonon occurs 
in other educational settings as well, but it may be more 
acute in military training environments where instructor 
changes occur frequently. That a similar adjustment was 
required in going from one CBE lesson to another was also an 
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interesting observation. In this case, h<;r/ever, it implies 
a lack of product uniformity standards during lesson develop- 
ment. Had such standards been employed, the lessons would 
have been more similar in format and would have required 
fewer adjustments on the part of the students. 

We would also note that the comments in this section 
concerning the active exploration of the characteristics of 
PLATO instruction stand out^ in strong contrast to the, 
soporific reactions of students to lecture presentations. 
Even the effort of the students to "beat, the system" may be 
viewed as positive when compared to the lack of interest 
exhibited in the lecture classroom. 

Discussion . The authors conclude in this section 
that "control of the instructional material must be handled 
very carefully.", It cannot be assumed, they say, that all 
Students will b^e motivated to do'their best work. That is, 
given some control over Jesson sequencing, there will jalnays 
be^a few students who seek to "beat the- system". Furthermore, 
student performance has not -been shown to be universally 
better when lesson sequencing has been put under the ^control ^ 
of the student rather than ^.the computer. However, Seidell 
('1975) has shown that bigh-per forming-sfeu^dent^s can benefit 
from a degree of student' control over sequencing. Learner 
control may be most effective, therefore, when individualized^ 
according to performance record. For a review of research on 
the issue of learner control in CBE, see the article by 
Steinberg (in press). 

Conclusion . -We have previously noted our reservations 
regarding some of the aut,hors* observations, but we concur 
with their conclusions that (a)' PLATO did not have a major 
adverse impact that precluded its potential effective use in, 
technical training, ^nd (b) refinements in' implementation 
procedures cotfld reduce the negative impact of those problems 
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that were noted. We believe that many of these refinements 
could be made quite easily. 

Impi^ct on Existing Air Force Regulations and Guidance 

Since we are not part' of the Air Korce training environ- 
ment, most of the remarks of this section are beyond our 
sphere of direct experience. As close observers of the 
entire service test at Chanute, however, we do agree with the 
statement that the- use of the PLATO system as an instructional 
resource did not necessitate great changes in Air Force 
regulations, manuals, and procedures. However, our view of 
the service test ^leads us to underline the caveat that 
future Air Force PLATO efforts should be conducted in the 
environment of systemati.c , in-depth instructional system 
development. • - 

New Training Capabilities . ' * 

" It is true,. 83 the authors' indicate, that the main goal 
or Phase li was to get operational training underway as 
rapidly as possible. This orientation limited the use of 
PLATO to presenting low-level instruction in automotive 
fundamentals using instructional, strategies similar 'to those 
used in conventional programmed texts. There was little 
opportunity, in Phase 11, to attempt applications which would 
utilize the unique capabilities of the CB& medium. 

We agree that the two areas discussejl here as potential 
cost-effective. applications of the PLATO system (i.e., 
diagnosis/troubleshooting and performance testing) are prime 
candidates for achieving cost-effectiveness while' substanti- 
^lly improving the quality of training. We also agree with 
thextentativeness with which these recommendations are put 
fbrwar^d^ The successful development of any CBE courseware 
depends u]:>on a great many factors. Only by assessing the ' 




117 



Ill 



instructional materials to be produced, the environment in 
which they are to be used, the capabilities and experience 
of the instructional developers, and the, management of the 
production effort can" one begin to predict the chances for 
success. With the right combination of these factors, 
however, the hoped-for advantages of these CBE appl icatio.ns 
could probably be realized. 

Overall Summary and Conclusions 

In this section- the authors summarized the important 
results of this extensive chapter on instructional impact. 
The major findings can be summarized as follow;^: 

1. The impact of PLATO on student attitudes was 
generally positive. 

2. Instructors reported positive attitudes toward 
PLATO initially, but a second' survey showed a decline in 
attitude. BaSed on more recent attitude oata (Klecka, 1977c) 
showing a generally ' favorable attitude toward. PLATO,- we 
offered the hypothesis that the negative attitudes found in 

a 

the second AFHRL-survey were due to_ temporary factors. 

3. PLATO had no significantly adverse impact on conduct 
of training which would preclude its potentially cost- 
effective use. Furthermore, refinements . in implementation 
could alleviate much of the negative impact that was 
observed . 

4. Existing ATC regulations provide adequately for'the 
utilization of CBE in operational training; no major changes 
in these documents are needed to incorporate CBE. 

^ b. Suggestions for additional PLATO applications which 
would utilize the unique capabilities of CBE were made. 

We concur in the^corfclusion that «'the overall impact 
of ?Lkl0^h33^\>e'eri more positive than negative. There is no 
r^^ason to believe that PLATO cannot prove to be an accept- 
able and useful training medium.'' ' . 

« 

N 
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VI. INSTRUCTIONAL MATERIALS DEVELOPMENT 

This is a well-written chapter providing a wealth 
of detail about the process and products of the lesson 
development efforts. Although headings are sometimes a 
bit misleading, the content is addressed in a logical and 
orderly fashion. 

Method 

No comment is needed. 

Results and Discussion 

Process description . It is indeed accurate to say 
that Phase II contrasted favorably to Phase I in^terms of 
the Vapid production -of effective lesson materials. It 
should be pointed out that both the methods used and the 
management staff changed substantially between phases. s The 
contribution of each to the enhanced efficiency is large, b.ut 
the relative importance of the components is, unfortunately, 
unknown. The model which did emerge during Phase II was a 
reasonable fit to the constraints, of the technical training 
environment. Future applications of CBE in a similar 3etting 
would do well to take it as a starting point and evolve 
further refinements. 

* The rtature of technical training . It is certainly true 
that, as this section states, the characteristics of the 
courseware development effort in Phase II were defined by the 
nature of Air Force Technical training. In contrast to 
higher education, where there is c6nsider<^able emphasis on 
development of the higher levels of cognitive processing and 
learning as an end in itself (i.e. ^'education" ) , the Air 
Kcrce technical trainee is taught only what he needs to know 
for apprentice-level entry into a career field. It is 
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assumed that further advancements will require further 
preparation in the form of Career Development Courses or 
On-the-Job Training. 

In the technical training environment, over-training is 
costly and is to be avoided just as is under-training. This 
philosophical. orientation toward training (known as the 
"lean approach") is realistic and effective, but it narrows 
the range of CBE capabilities which can be exploited* 
Whereas in higher education settings the simulation and 
modeling capabilities of CBE are usually considered to be 
quite attractive (because they can be us- 1 in developing 
higher-level cognitive skills), these features were not seen 
to be so important in the apprentice-level training 
provided in the Special Purpose Vehicle Courses.*^ Hence,' 
such sophisticatecJ applications as diagnosis/troubleshooting 
and performance testing (discussed in Chapter V) were not 
at^tempted. 

CAI Lesson Development , 
The model s We agree that the procedural model 
described in this section is a generalized composite of the 
process actually used. Although individual deviations from 
it were common, the description provided by the authors 
conveys the general approach that was followed. The reader 
should note tha^ th? report authors abstracted this descrip- 
tion, from a series of interviews with service test personnel. 
No formal delineation of procedures had bee,n made by the\ ISD 
staff. 

Central feo the model is the concept of a team approach. 
In contrast to the pattern followed during Phase I in which 
each individual author was given sole responsibility for 
all phases of lesson development, two lesson production 
teams were established in Phase II . Each team included 
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two to three of the original authors, but some of the 
development tasks were shared by individuals who had more 
specialized duties. That is, an instructional programmer, a 
computer programmer (i.e., a "coder"), and a subject matter 
expert supplemented the authors' efforts. The authors had 
had previous experience with CBE by virtue of their Phasfe I 
work, but the other team members (including the instructional" 
programmers) had only had experience with conventional 
programmed instruction arfd stand-up instruction — they had not 
previously developed lessons for a CBE system. 

As stat€d in the AFHRL report, the procedural model "^s , 
a description of the processes followed in step H (i.e., 
"Plan, Develop, and Validate Instruction") of the overall 
model for instructional systems development (Figure 11). 
The first three steps of the overall process had already 
been completed by the ISD, team prior to the start of Phase 
II activities. 

The nine steps in the lesson development process are an 
adequate portrayal of the procedures actually followed. 
General descriptors of the steps are: 



„1. 


Lesson selection. 


.2. 


Research, 


3. 


Lesson planning. 


4. 


Off-line lesson preparation. 


5. 


Un-line coding. 


b. 


Rre-tryout review. 


Y. 


^tudent tryout. 


B. 


Validation, 


9. 


Finalization. 



A more complete description might have indicated the* 
nature and extent of individual deviations from this model. 
We also would point out that in the lesso.n planning stage 
(step 3) the only optional media used at Chanute were 
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.rear-projected microfiche images. Touch panels and random- 
access audio devices were not used in any way during Phase 
II, to our knowledge. Francis (1976a, p. 66) provides a 
survey of the use of peripheral equipment at ARPA/PLATO 
project sites. 

It is not apparent from "the discusfeion • of the model 
that step 8 was separated^f rom step 7 'by a period of up to 
six months, but the authors do make this' point later in 
the chapter. For more information about lesson development 
■ procedures in Phase II, the reader is referred to the entire 
first chapter of a report by Klecka (1977b). 

• Assessment of the model . This section gives a further 
elaboration of Chanute's courseware development model and 
assesses its strengths and weaknesses. While a more complete 
analysis would be useful for the sake 'of future ' development 
efforts, the information provided does give a general over- • 
-•view and is consistent with space constraints. 

Many of the problems noted relate to the implementation 
..of the model at Chanute and are not disadvantages inherent to 
the model itself. The problems with communication, for 
example, developed because of the sudden manner with which 
the Phase II effort was initiated. The PLATC authors were 
'put under the direction of the ISD team af a time when the 
ISD group had already completed most of the initial planning 
for the course revisions. The PLATO authors had not partici- 
pated in this planning and the ISD group had had little 
previous experience with CBE (spme of the 'iSD personnel had 
attended a portion of the TUTOR language workshop at CEHL two 
years previously, but had never done any direct CBJ: lesson 
development). Many basic issues in philosophy and orienta- 
.tion had to be settled before the effort could get underway, 
and the process was complicated by the pressure of a tight 
production schedule.' These problems in communication would 
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not have been so imposing if all groups had had adequate 
training and if all dnitfal planning could have been done 
^Jointly. ^ » . , . . 

Because the PLATO authors were the only, team members 
who had had previous experience in CBE, they retained a 
central role in the Phase II effort. Their role was' nb^ - , 
drastically different under Phase II than it had been unde'r 
Phase I. They still had primary responsibility for most 
aspects of lesson development, but relied on th-e instruc- 
tional programmers and subject matter experts as resource 
persons and for quality control. In future applications, 
whene^more care can be taken in training and in assignment" 
of roles, a more efficient team organizatiin should be 
possible. 

The development .of the student router, test administra- 
tion routines, and other course management software followed 
a- different model as indicated in the report. For a detailed 
description of the uses of 'the PLATO system as a management 
tool at Chanute, see the second chapter of Klecka (1 9.77b). 

Appendix I 

. Because of the importance of Appendix I to the under- 
standing of the conditions and procedures'* in the Chanute 
PLATO project, thi.s critique devotes ..a section to comments r 
about it. The appendix consists of several parts',, the most' 
important, of j^hich are the scenario and the samp^le lesson. 

r Scenario. The appende'd scenario is-a livelier and 
more picturesque view. of the lesson development steps 
described in Chapter VI." As such,' ijt is a very vgluable 
addition. Since any scenario can consider only a limited 
number of situations if it is to be of reasonable length,, 
-ther-e^is--the-possibi-l-ity;:that the reader will- gain an inborn- " 
plete perspective of the project. To avoid that potential " 
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problem, we have included below several additional consider- 
ations that should broaden the reader's view. 

Some authors suffered more from morale problems ,than did 
"Mr» Arthur". Authors, frustrated by the^ constraints and 
demands of their environment (e*gJ, terminal shortages, 
uncertain project goals, lack of " administrati^ve support) 
formed constantly shifting alliances with and against each 
other, the ISD^group, ttie evaluation staff, and th^ tradi-,^, 
tion&l course personnel. \ 

Because of the author attrition noted, quite a number^^T^ 
lessons were written by one author and, after running- a /ew 
students, revised by another. This may explain some of ^he^ 
inconsistencies within lessons noted previously and in the 
following section* 

Sample lesson . We would encourage all readers to read 
through carefully these 'sample frames from a representative 
lesson • They-^exemplif y the comments made in this chapter in 
very concrete terms. From our perspective, the lesson 
selected is typical of those produced during Phase II. 

Strengths and weaknesses are found, not only between 
lessons, but also within them. In nnany cases the quality 
of one aspect of a lesson varies widely from frame to frame 
within that lesson. For example, in the clutch lesson de- 
scribed, the graphics are very well dene, but are not always 
available when most needed. Frame 39 gives a long "knee- 
bone- is-connected-to-the-^leg-bone" xlescription that desper- 
ately needs graphic support (preferably with animation). 
For a second example, the multiple choice questions within 
the body of the lesson are reasonable and integrate well 
with the presentation, but the choice of matching items in 
the MVE includes both tangibles (copper wires) and intangi- 
bles (solid friction) as well as s.'ngular and plural forms. 
Both examples demonstrate the variation in lesson quality 
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we have observed in the Chanute materials ^nd seem to indi- 
cate that ^the potential for higher quality lessons lay 

almost within the project's grasp. 
«» 

Courseware Development Issues and Constraints ^ 

Prior lesson development efforts . ^Ln this section the 
author:: give a general description of the activities, decisions 
and, outcomes of the first two years of the project, i.e.. 
Phase I. During this period the project's goals were either 
undefined or were changing rapidly—it wasn't until the 
beginning of Phase II that a cohesive curriculum development 
effort was begun. The lessons that were produced in Phase I. 
were developed under ttie "Autonomous Authorv" approach. 
Since this basic model had been used successfully at the 
University of Illinois, CERL ha.d advocated initially that ^ 
individuals be given full responsibility for developing 
lessons, i.e., to act as independent- authors. Unfortunately, 
this approach was not,^well-sui ted to the Chanute situation 

r 

because several key prerequisites for its success (i.e., 
self-selection of authors, thorough subject m.atter knowledge, 
relevant instructional design experience) were absent. Later 
efforts by CERL to foster a team approach were unsuccessful 
because of the flbctuationi of project goals. 

Although, as noted, self- training ^ in programming 
techniques occupied a great deal of time during Phase I, we 
see this less as sn .actual need and more as the authors' 
attempts to "do something useful" during a period when other 
goals were rapidly changing. 

The diagnosis that higher management had little interest 
and provided little support to the PLATO project because of 

-its -research orient^t i-on-l-S"Consrstent" wrth our .observation's ~ 
and it seems to be an important problem for future sites as . 

-well. The use of a novel medium such as PLATO suggests the 
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need for a-research orientation to examine the- potential of 
the medium, but a research orientation'-may seriously 
jeopardize the acceptance of the new medium and limit the 
commitment toward it on the part of higher-level admin- 
istrators. " * 

. The conclusion that Phase I resulted^in a fragmented 
non-curriculum not conforming to civilian or military 
standards for training is congruent with the findings of the . 
May,. 197^, review committee. 

Project management . This section includes an excellent 
description of the differing expectations of the iiJD 'team and 
t'he author group. During the transition to Phase II, the 
authors felt they were being told to sacrifice quality for 
high production; the 'iSD staff felt the authors were unmoti- 
vated. On many issues, the two groups' views were vastly 
different. Initially the ISD staff hoped to produce PLATO 
lessons as quickly as programmed texts (50hr/hr). The 
authors were used to working at a- rate about an order of 
magnitude slower than that (^400hr/hr). Both estimates 
'(obtained through CERL interviews) were unrealistic and 
conflict on this point was inevitable. It is significant 
that problems such as this were solved or circumvented 
quickly so that courseware production could proceed. 

Other management problems arose because the use of CBE 
was seen by course ^personnel as being a temporary experimental 
project having little place in operational training. Future 
applications should seek to utilize CBE as an important" 
resource rather than treat it as an artificial and short- 
lived experiment. If.' this can be done, it \should be possible- 
to circumvent many management and implementation problems. 

At the end of this generally excellent "section , the 
JlEHRI^repo r-t—ind-iea t es tha t-«-[-b-} e-in«— H-near-^nd^-e^n-r"th e~ 
lessons did not tax the capabilities of the PLATO system." 
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The distinction between "taxing" and "exploiting" the vari- 
ous powerful features of the PUATO CBE system was confused fre- 
quently during Ch&nute's history. Members of CERL's MTC. staff 
and some members of the Chanute author staff regularly advo- 
cated the exploitation of sys^tem features for sorting student 
response data, providing feedback for specific wrong answers, 
and for judging the adequacy, of a. wider variety of student 
responses. This advocacy was often understood to mean "the 
PUTO system has clevier and unique features .which ought to 
be used because they are so uhusual, because they 'give-the 
system a workout' (as if th^. system somehow grew sluggish if 
not regularly exercised) , or because the use of these fea- 
tures justified and acclaimed th^e PLATO system." In fact, 
those encouraging the use of tliese features saw instructional 
or administrative problems which had potential solutions 
available within the system itself. We make this comment 
to illustrate the .nature and extent of the problems caused 
through differing concepts of the role-, and capabilities of 
the computer. Hopefully, these conflicts can be' circumvented 
in future development efforts through adequate training. 

Hardware and software related constraints , '^e would 
agree with the assessment that the site was not restricted by 
the features and ease-of-use of the TUTOR language. V'hile 
it is true that some of the authoring staff had limited 
programming ability, others were quite capable and most 
programming needs could be handled without outside assist- 
ance. The presence of "coders" (i.e., persons whose primary 
role was programming the. computer rather than' the development 
of teaching sequences) on the staff greatly enhanced the 
effectiveness of the site. 

For a variety of r ea s n ns^i-tm3^4-iH-^g-t4ie-fac1r-th^1r-j:tr^as 

one of the first remote PLATO IV sites Chanute did suffer 
from hardware problems to a greater extent than ,^ny other 
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ARPA site. The^ shortages' and" delays associated with terminal 
• and microwave inrstallation were especially detrimental to 
' Chanute. Other resource shortages (i.e., in ECS, disk file 

space» and computing power) were experienced by all users and 
"hindered the lesson development process at Chanute as well. 
Fortunately, most of these problems have been alleviated' 
through various hardware enhancements to the central process- 
ing system. • - , 

Courseware development . Chanute was one of the earliest 
users of microfiche. As such, they spent a great deal of 
time during Phase I exploring and defining the limits of the 
microfiche system. Nevertheless,' it seems an exaggeration to 
-state that several man-months were spent during Phase. II in 
refining this process. In any case, the story ofi-rChanute' s ' 
use-pf microfiche is so extraordinary and enlightening that 
Francis:, (1976a) devoted a- spb-chapter to describing it. 

Other problems described in this .section relate to 
the fact that- the authors were not true subject matter ex- . 
perts (so that they had to spend time in- researching the 
content to be taught) and the ISD policy of limiting the 
amount of student testing a'le§son was to have prior to its 
operational use. These problems should be addressed in 
planning for future 'CD£ deyej-opment efforts. 

We agree heartily 'that there was a problem in determin-' 
ing.the proper deptK of training for an apprentice mechanic; 
this resulted in some appan^nt inconsistencies in the Chanule 
lesson materials. For example, the basic electricity lesson 
delves fairly deeply into theory, teaching mechanics 
trainees about atoms and electrons, but , less than an hour is 
spent in the only lesson describing the complexities of 
_emissjj^n_Ggmtrol^ p valves. 



Part of the confusion may lie ip the fact tF^at the 
Specialty Training Standard (STS) for most knowledge items 
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' ithe material taught by CBE) indicates that students should 
- be- taught to the "B" level. The "B" level is defined as: 
"Can explain relationship of basic facts and^state general 
principles-about the subject. (PRINCIPLES)" Althougn the 
cour^e''documents specify the depth of training to be ?.t the 
"B". level, the lessons which we're produced seem to us to be 
aimed at the "A" level. The "A" level indicates a more,, 
superficial level of training and is defined by: "Can 
/ identify basic facts and terms ^abo'ut .the subject! (FACTS)** 
Since field , results (reported in Chapter III) indicate 
acceptable student performance, yet the teaching and testing^ 
are not at the "principle" level as defined (see Appendix I 
of this report or Klecka, 1977a), it may be that training at 
the "A" level is sufficient. This discrepancy between train- 
ing levels, plus the problems noted in the AFHRL report, 
gave rise to the "frequently, expressed concern about 
insufficient depth of training" voiced by the authors. 

Authoring . The discussion in this section further 
illuminates some of the problems felt by authors under the 
conditions of the service test.- Attention to ways of cir- - 
cumyenting these problems should be given in future efforts. 
Since the contents of the Lesson Characteristics , 
^ Instructional Strategies , and interaction sections are so 
directly related to the CEKL publication by Klecka (19YYa), 
and because the elaborations .anjd added emphases of Chapter VI 
are consistent with our observations, it ics not necessary for 
us to comment further . ^ 

Instructional programming features . Although many 
Chanute lessons used microfiche, it is remarkable that few, 
if any, 6ver used a microfiche ^as part of an end-of-lesson or 
block test. Similarly, although students were shown many 
microfiche, rarely were they asked any questions about them 
during the lesson. The effect of this unusual use of 

^ V .* ' * 
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microfiche is that th^' information presented via microfiche 
seeTms. related , but less important, than infol^tnat ion presented 
in other ways. It is stated elsewhere in the AFHRL report 
that students found the. use of m.icrofiche images very moti- 
v^ating; it may be that motivation was the primary objective 
and was' achieved. - * ^ ' * 

Graphi'^.s, typically, were n^ore completely integrated 
into the Wesson. As this section of Chapter VI observed, 
slow-plotting graphics are disliked by students. In some 
cases overly^complex^' screen graphics were used when 
microfiche might have been employed more. inexpensively . In 
one documented case (Francis, 1977a), one of the best and 
most hard-working programmers at Chanute spent two weeks 
copying a singje (slow-platting) display of an electrical 
schematic. 

The last' point we would like to make about this section 
is to re-emphasize an AFHRL observation which we feel summar- 
izes our main criticism -of the lessons. Students reportedly 
complained that test questions did not say thin'gs in the same 
way as the text in- the body of the lesson^ We agree that 
there is evidence to suggest that the students were merely 
associating words, not learning concepts. 

' Student evaluation . No corn'ment is needed. 
Lesson validations ' No comment is needed. ^ ' 
Validation techniques . This section also closely follows 
Klecka (1977a) and requires ^only one further comment. Among 
the "technical errors" mentioned at the ^nd of this section . 
were a' sizeable number of lesson execution errors. Although 
these ernprs are carefully documented by the computer, 
several lessons which had passed^ l:he validation criteria " 
continued to cause this major kind of error as long as nine 
months after validation. This suggests a very narrow view of 
"validation". . ^' ' ' 
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Conclusions 

We concur with the conclusions that: 

1. Phase II lesson development procedures were more 
effective aud efficient than those used in Phase I. 

2. • The Phase II lessons did not exploit many CBE capa- 
bilities because of inadequate resources and a lack of 
relevant training'. 

3. .Role specializatio-p contributed to increased effic- 
iency of lesson production (4. e., we, too, -expect- a team 
specialist approach to emerge as most appropriate for 
technical training applications). 

4. The PLATO syistem provided ample flexibility in ~^ 
design strategies and data collection. 

5. Sophisticated CBE capabilities may not be necessary 
for lessons having limited (i.e., low-order) objectives. 
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Chapter VII. MANAGEMENT OF THE PL'ATO BASED COURSE 

•- ^ 4 

We find thi^ whole chapter rather complete and percep- 
tive. We expect that future development efforts can profit 
greatly from the discussion it provides. Due to^ ^pace 
considerations, our comments are directed only at the 'latter 
sections ''of the chapter- 

Management of instructional material . Two items related 
by Chapter VII deserve additional emphasis. We agree with 
-the -suggestion that- it is not wise to automate a function 
merely because it can be done or because it might save a 
small bit of time. It indeed seems likely 'that student and 
instructor motivation and morale might slip if certain 
scheduling, monitoring, and controlling functions w6re left 
entirely to the computer. The next section of the critique 
discusses some problems that' were pointed out to us concern- 
ing the new role of the instructor. 

The AFHRL report makes a comment that should be reflected 
back to the operational' problems noted and to the results of 
the cost-effectiveness evaluation when it observes, "Lt]he' 
course was operated under the assumption that PLATO" would not 
reduce instructor requirements and therefore the system 
should be utilized only in ways that** fit into traditional 
modes of instruction." Although such constraints are 
sometimes necessary, they force assumptions and allow 
interactions which drastically change the environment of 
implementation. In our view, the predominant attitude at 
Chanute was, "We will permit the use. of PLATO in^'those places 
where we can -force it in to\raditional roles " rather than, 
"We will adapt our methods and procedures so as to take 
maximum advantage , of the p6tep.tial of PLATO." 

Instructor role in PLATb-based training . The AFHRL . 
report draws a somewhat unpleasant portrait of the role, of 
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the instructor^ teaching a block^ of the automotive course 
after it had been modified for PLATO by ISD. This picture 
is, unfortunately, supported by the attitude data from Chap- 
ter V* Two bases for di'scontent emerge from the discussions 
in* this chapter . \ 

First, the lessons themselves were a problenu^By^'^ 
measurements, these lessons were not-."po3Li-sHed" , i.e., thf y * 
required the constant presence of a knowledgeable proctor- 
to assist students/having difficulties. .Nagging, problems 
kept cropping up for the instructor. And, as described in 
this chapter, efforts to correct those problems were thwarted 
by an unfortunately complex set of procedures. (Recent addi- 
tions to the PLATO system have provided a convenient way for 
lesson problems to be reported to the author staff without 
entailing the "red tape" noted in the AFHRL report and 
without causing major interruptions to the work of the PLATO 
staff. Chanute personnel have implemented this system and 
instructors are using" it.) 

A second problem was described by the heaC of ^the ISD. 
group (personal communication) as the reason why group-pacing 
is. worse for 'instructors than either self-pacing or lock-step. 
He explained that because the slowest student sets the rate, 
the instructor cannot sit down to review and^ carefully teach 

the slower students--that would generally mean the group 

< 

would move at a still-slower rate. Instead he can only prod 
the student, or sit with the slow student and give him the 
answers so the group can proceed. With self-pacing, the 
instructor can help the student by individually tutoring him, 
by providing additional materials to , fill missing prerequis- 
iteSf-or-by—re^tarting the student at the point at which he 
found difficulties. Lock-step pacing usually means stand-up 
classroom instruction with the • instructor as the center of 
attention and clearly in command. With group-pacing, the 
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slowest^student provides the basis for proceeding to the next 
topic; in many ways the instructor is peripheral or superflur 
ous to the activity taking place. 

The last part of this section points up a t'ypical prob- 
lem of implementations: the conditions under which the 
materials were validated are not the conditions present when 
the materials are used for operational training. There is a 
significant threat to the integrity of the validation because 
students are now allowed or encouraged to .take notes for use 
on the end-of-lesson tests which were used as the validation 
criterion. While it may be that note-taking is an important 
learning strategy, the effect of using notes during the test 
is likely to be significant. Furthermore, CERL staff observ- 
ing in the Chanute PLATO classroom after the lessons had been 
validated saw students "getting answers" by questioning other 
students or glanci^ng over at a neighboring terminal. Test 
performance under these circumstances may differ from that 
under the conditions originally planned for the lessons 
(I.e., no notes- and each student doing his own work). 

In general, the report authors have pj^ovided valuable 
insight and have made excellent recommendations regarding . . 
actions to be taken by those in charge of maintenance and 
operation of the special purpose vehicle course. The Chanute 
staff would do well to heed these recommendations. 

Management of student testing . This section accurately 
depicts the various capabilities of the PLATO system and its 
utilization by the Chanute author and instructor staff. For 
greater detail about instructor comment storage and retrieval, 
consult the chapter by Klecka (1977b) dealing with manage- 
ment applications at Chanute. 

General Conclusions Concerning Management 

The conclusions presented are an appropriate summary 



134 



128 



for this chapter.' One sentence is especially interesting: 
the report says "[wjhile different techniques were necesssry 
to allow for PLATO'S idiosyncr acies , these techniques served 
to facilitate efficient training rather than impede it." 
This is a familiar finding for many management tasks under- 
going computerization. An explicit list of examples of such 
techniques would have been especially valua'ble. 
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Chapter VIII. fjuhaN FACTORS ;NALYSIS 

From the view of the specialist in Human . Factors , this 
chapter is somewhat disappointing in its I'ack of quantitative 
data. For example, it would be interesting to have specific 
data for the exper ience . at the Chanute site on terminal 
reliability, in addition to the CERL-provided data on system " 
reliability. The reliability information provided by CERL 
in6ludes terminal reliability data averaged over the entire 
system. Being located only about 15 miles from the CERL 
hardware maintenance facility, the Chanute sice's Vxper- 
ience may actually be better than the system-wide average 
reported by the authors.. ^ 

From a practipal standpoint, however, it seems clear 
that all major considerations of systems design have been - 
evaluated fbr potentially fatal limitations and that no such 
limitations have been found. Given the limitations of 
resources under which the evaluation was 'per formed , the-, 
coverage on this point seems entirely appropriate to the 
needs of the Services. 

One point that may need clarification is the recommenda- 
tion for indirect lighting to prevent -glare frcm the f.ce of 
terminals. Many people include diffused light as a form of 
indirect lighting. Strictly speaking, they are correct since 
the source of light itself is not visible. However, glare 
from translucent d if fusers- can be just as troublesome as 
glare from exposed bulbs or tubes. The only sure cure for 
the problem of glare from display screens is to insure tfat a 
dark ceiling is the only thing visible by reflected vision to 
one seated at a terminal. This may be achieved by -sharply 
directional lighting deeply recessed in the ceiling, reverse- 
angled or vertical screens, or a combination of the.se 
approaches. . 
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Chapter IX. EXPERIMENTAL STUDIES 

The experimental study reported in Chapter IX is the 
onl^y one of seven studies planned for ,fhase III that was^ 
completed. The "Historical Perspective" section of Chapter 
II indicates that it was originally planned to refine each of 
these seven, studies through three separate dterations. The 
material reported in. Chapter IX represents the one and only 
iteration of the Technical Order (TO) stucj^ that was com- 
pleted. * ' ' 

Lesson Sequence and Development Time 

The description .given of the games used in the TO instruc 
tional sequence is accurate.' It is important to note that 
the strategies used in these games are sufficiently general ' 
as to allow them to be used to teach some other subject 
.matter. That is, -there is nothing inherent in the strategies 
which resti;icts their use to the TO material. 

, It is difficult to say how an automotive' repairman 
trainee might perceive these game$. To CERL evaluators, 
however, they appeared to be either so complex that learning 
the rules and mastering the game was more difficult than 
mastering the material being taught or that they were lack- 
ing in motivational power. Readers should note the complex- 
ity of the "Wizzard Game" deseri^bied^ and illustrated in this 
section of the report. Not only is the game complex but the 
rules lack any obvious relationship to tasks involved in 
finding and using TOs . 

CERL personnel have verified that the potential for the 
use of gaming strategies is high for TO instruction. An 
easily learned game requiring the participant to classify TOs 
in correct order was written by, CERL personnel. This game 
was interesting and ^challenging even to those who had no 
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intere^Nn TOs. Several PLATO authors played this game 
simply for, its recreational value and became facile in 
ordering lists of TOs in -the process. This game was not used 
m the TO sequence because it was decided by those directing 
the study that it did not meet the design criteria that had 
been specified. 

The evaluators in charge of the study are to be commended 
for attempting to measure the development time expended for 
•each of the game lessons and the mainline instructional 
lessons in the TO sequence." Data of this sort are scarce but 
are of obvious importance in answering questions of cost and 
efficiency of production methods. Future development efforts 
should attend to the problem of collecting these types of 



data. 
Method 



Evaluation design. The use of three conditions— conven- 
tional non-PLATO instruction (NP), PLATO-based version 
(PV), PLATO with adjunct gaming (AG)— was twofold. One pur- 
pose as reported, wSs to test the efficacy of gaming strate- 
gies for improving student performance and attitudes. The 
other was to see if PLPTO-based instruction by itself could 
accomplish this result. Conventional instruction in the use 
of TOs had long been recognized at Chanute as being ineffec- 
tive. Thus, a complete set of PLATO lessons was developed to 
tench the objectives of the conventional TO sequence. The 
games for the adjunct gaming condition were then appended 
onto this set of lessons by interspersing them among the 
instructional modules as shown in Figure 12. 

The fact that the games wer;e a'dded onto a sequence of 
simple tutorial PLATO lessons,- which were essentially com- 
plete in themselves, was not the most favorable environment 
in which to determine the value of gaming strategies, In 
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this 'context, they are simply an appendage which would 
certainly lengthen the time needed to complete the sequence. 
Moreover, in the event that student performance, was already 
high with the unadorned PLATO TO sequence, any improvement 
coming as a result of the games would be diff}.cult to detect. 

Measures . The reporting of the^actual attitude ques- 
tionnaire would have been useful for those who might wish to 
use it or make independent judgments regarding its validity. 
Additional information concerning the reliability of test 
scores and attitude scales is presented in a later report 
section. 

Procedures . The discussion of procedures suffers 
somewhat from lack of clarity. This, no aoubt, is largely 
due to the complexities caused ,by'the differing schedules 
for the general purpose and special purpose courses. Such 
complexities are to be expected when research must be con- 
ducted, in an operational environment. 

Results ^ 

Lesson performance findings . There are two main 
difficulties with the analysis of procedures used for the 
lesson completion time and lesson score data.^^ First, it is 
probably inappropriate to have conducted univariate tests on 
each of the seven modules for reasons stated earlier in this 
critique. Second, a more sensitive measure probably could 
have been had by combining lesson time data and the lesson 
Sycore data across the seven modules. Given the distribution 
of mean times and scores shown in Table 3^, however, it is 
not clear that even the overall time and lesson score meas- 
ures would have de':ected a difference between the PV and AG 
conditions, 

Attitudinal findings . The PV condition students had 
significantly higher attitudes toward TOs than did the NF and 
AG students. 
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Disoussion 

The games introduced into the TO . instructional sequence 
were simply appended to the established lesson sequence. One 
would expect, therefore, that these games would lengthen the 
completion times of these lessons. That such a result was - 
found comes as no surprise. The fact that the attitude 
towards the subject matter was poorer for the students in 
both the AG and the NP conditions than for the PV condition 
is rnteresting. It would seem th&t attitudes were improved 
by the simple use of PLATO but that this advantage was 
negated by introducing games into the sequence; This seems 
to verify our opinion that the games were amot ivational and 
dull. Without being able to view the questionnaire and 
because of the acknowledged" problems with the internal and 
external validity of the study, such conclusions can only be 
put forth tentatively. 

The AFMRL evaluation report wisely refrains from general- 
izing from the study's 'findings on gaming. Had the study 
been able to proceed, it would have gone through two more 
iterations during which the games, lessons, a'nd data collec- 
tion procedures would have been refined. Such refinement 
might have included integrating the games into an instruc- 
tional sequence instead of just- appending them to a sequence 
of PLATO lessons. In this way a more valid test of the 
efficacy of gaming strategies could have been obtained. 

hinally, it is important to note that -the PLATO ver:?ion 
of the TO lessons accomplished much of what it was hoped tne 
adjunct games would. These lessons, without the games, saved 
student time and produced more favorable student attitudes 
toward the subject matter. In fact, we feel that the conclu- 
sions of this chapter overlook (or at least underemphasize) 
one of the most important findings of this study. Chanute 
had a long-standing problem (i.e., the effective teaching of 
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Technical Orders) which had resisted several previous 
attempted solutions.* The results of using PLATO strategies 
without the games (higher learning efficiency with better 
attitudes) suggest that PLATO may be the lohg-sought-af ter 
solution for this problem. The ov.erall conclusions of' 
Chapter IX have a tone of failure, although from the view of 
a student, instructor, or course administrator, the results 
are positive. 
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Chapter X. OVEi^ALL CONCLUSIONS' & RKCOMMENDATIpNS 

/ ' 

Overa] Conc lus ions ' . ' 

The overall conclusions adequately summarize the findings 
af the previous chapters and contain no surprises. In theyir 
brevity, they overlook a few problems (such as declining 
instructor attitudes) and a few opportunities (such as the 
potential for computer-aided test del ivery ) but concisely 
remind the reader of the salient outcomes, v They are restated 
here (in abbreviated form)- for the convenience of readers 
who do not have access to the AFHRL report. 

1. The PLATO system was an effective medium for present- 
ing -task-related cognitive materials. 

* 

^ 2. The Chanute PLATO application was.-' not cost-effective 
when compared to courses employing less sophisticated indi- ' 
vidualized media to perform similar functipns'in this 
application. However, as .shown by our comments on Chapter 
IV, the Chanute application may have approached cost- 
effectiveness in spite of the fact that it may not have 
required^the full capabilities of CDE. 

3. PLATO did not have a large impact on the main fea- 
tures of the -training environment; it was acceptable to stu- 
dents and instructors and it was compatible with existing 
organizational structures and ' controls . However, little in 
the way of new training capabilities was demonstrated. 

-4. Due to the complexity of the courseware preparation 
process, greater efficiency resulted from a team approach 
employing role specialties, than from the single instructor/ 
author approach. 

5. The Chanute effort was limited in attempting addi- 
tional applications by constraints in management, resource 
acquisition, personnel training, and experience. 
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6. Som^ management difficulties were experienced 
because the PLATO system was not an Air Force resource. 
Adjustments were necessary in the areas of training activity 
scheduling and course instructional design selection. 

?• Few usage problems were present in terms of student 
interaction or courseware authoring. PLATO was found to be a 
reliable tool oncfe system expansion and exper ime'llitation was 
moderated . 

Conclusion #6 requires a comment. It implies that 
certain management difficulties caused by resource constraints 
could be eliminated by Air Force/military control of a PLATO 
system. Although maintenance time, for example, could be 
shifted to accommodate Chanute^'s needs, adverse effects might 
occur for other users (military and civilian) in other time 
zones. Nevertheless, it is realistic to >expect that any 
agency large enough to support its own system would want to 
do so in order to gain control of scheduling, maintenance, 
and resource allocation parameters. 

Recommendations Specific to Chanute 

Chanute CBE operations . The seven recommendations on 
thi's topic are restated here in abbreviated form: 

1. In this- recommendation the authors state^ that 
"PLATO applications should be selectively expanded to other 
course area^s where CAI capabilities might be used to solve 
training problems, increase- instructional efficiency, reduce 
training resource requirements, etc." They also suggest roles 
for specific organizational units at Chanute and advocate the 
development of a detailed plan-* to guide expansion of PLATO 
applications. 

2. The authors *-ecommend the self-pacing of the shred 
areas cf the 5PVC to take advantage of CBE's individualized 
nature, limit the need for enrichment materials and provide 
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for more efficient use of available terminals. Although this 
recommendatiprr could lead to additional time savings, it 
would also entail purchase of additional, training equipment. 
Nevertheless, it seems .useful to prepare, an analysis of 
projected costs and savings . 

3. The authors recommend that terminal utilization be 
maximized by shifting surplus terminals to other courses to 
enable exploitation of cost-effective applications. 

4. The report writers state that "Direct communication 
o * between authors and instructors should be promoted to 

facilitate application of fLATO, and increase instructional 
^efficiency." Perhaps increasev^ use of the recently added 
"TERM-comments" and "lesson notes" features of the PLATO 
system will help meet this need. 

5. This item is' a recommendation to install indirect 
lighting in the CBE classroom to eliminate problems of glare. 

,As indicated in. our critique of Chapter VIII, directed beams 
of vertical light are preferable to indirect light In "tTil%-"^ 
case. ; 

6. This item recommends a reduction in the instructor 
force in the - common 'course segment of theSPVC or some use of 
less expensive instructor aid personnel. 

^ Some additional remarks seem in order regarding recom- 

mendation #1. Though we don't have the appropriate back- 
ground to comment on the roles suggested for the various 
organizations noted^we concur with the general approach. We 
agree heartily that "PLATO applications should be selectively^ 
expanded to other course areas where CAI capabilities might 
be used to solve training problems." We believe that manage- 
ment and instructor acceptance of PLATO might have been 
significantly enhanced if it had been introduced as a tool to 
solve a training problem perceived by staff in both groups. 

Similarly, the effectiveness benefits might have 
outweighed the costs if the chosen course had contained 
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unsolved problems relating to training effectiveness or train 
ing' expense. Chanute attempted to squeeze higher -efficiency 
from a- training program which was already reasonably opti- 
mized., The sophist ica ted and expensive technology that they 
used to do this required that they'realize substantial gains. 
Chanute staff shojjld not be criticized, ^however, for the 
choice of a course to implement PLATO, nor for their decision 
to use it as the central medium for teaching. The many, 
.constraints on them, ^lus the, conventionaT wisdom of that • 
time .dictated these factors. However, in a document making - 
recommendations for futjjre projects and sites, it is very 
important to stress the application of CUE for attacking 
unsolved training problems rather than as, the "medium of 
preference" for implementing all new instructional materials. 

Development of CAI materials . Five recommendations 
regarding courseware development and validation are offered. 
They are restated here for the reader^s convenience. Accord- 
ing to the authors: 

1. The PLATO micro ''iche procurement process should be 
analyzed, then improved for greater efficiency. Toward this 
end, a new microfiche camera is being developed at CERL. 

2. Redesign of the microfiche projector might eliminate 
focus and registration problems. 

3* CBE courseware should be developed through a team 
approach using specialists who have overlapping knowledge of 
other team member's roles in the lesson development process. 

4. A training program should be provided for individ- 
uals who perform courseware development functions. This pro- 
gram would emphasize instructional programming concepts for 
preparing individualized materials. 

5. Instructor and supervisor involvement should be 
increased in thel review and 'revision of PLATO courseware. 
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We concur that a training program for instructional 
design is needec for courseware developers We also feel 

that training for the managers of the developers is needed. 
When the Chanute project was started, no such training was 
av.ailable; in fact, the knowledge base from which to train 
staff was largely incomplete. With the experience gained 
from the implementation of PLATO IV, it is now reasonable 
to begin this task. 

It is not clear whether the order of the recommendations ^ 
indicates their perceived importance. We would rank the 
suggestions for an instructional design training program 
a team approach to courseware development (//3) and 
instructor/supervisor involvement (#5) ahead of the sugges- 
tions about microfiche redesign. 

PLATO Applications 

Suggestions for how to use PLATO in future projects are 
some of the most itnportant results to be delivered by an 
implementation/research project such as this. The applica- 
tions suggested by the authors appear to have potential for 
cost-effectiveness.' A more extensive list of potentially 
cost-effective uses for CBE is contained in a list of CBE 
site management guidelines (Francis, 1977b). 

Some Final Remarks 

The comments by the authors in this section are a 
fitting summary to a well-written report on an extensive 
undertaking. We .-ould like to add a few general ^remarks . 

Readers of this (and other) reports of the use of CBE 
systems shoUld consider that the results obtained are rather 
sensitive to the conditions of* the implementation. Variations 
in the attitudes of the instructors, the quality of the 
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lessons, student-pacing modes, and the integration of CBE 
materials with instruction presented by other media can 
strongly influence outcomes and results. 

Although some of thB situations described in the report 
are necessarily related to conditions found only at Chanute 
or only with the SPVC, in general, the problems encountered 
are not uncommon or improbable, nor were the solutions used 
atypical or unlikely Jto be tried elsewhere. 

We would like to take, this opportutiity to again commend 
the project evaluators and the writers of this report for a 
significant accomplishment of a difficult task. One of the 
main benefits of the Chanute research is the extensive data- 
base it generated. Supplementary anSlyses of this data are 
underway at CERL and are expected to lead to additional 
reports on CBE and related topics. 

While we concur with the third of the overall conclu- 
sions of the AFHRL report (i.e., "[l]ittle in the^s^ay of new 
graining capabilities was demonstrated"), we suggest that 
re^\^tlei:^of the AFHRL report not view the Chanute experience ^ 
as a^n example of how training can be structured around a CBE 
system to .take maximum advantage of its" capabilities , but as 
an example of how PLAT'O can be incorporated into a traditional 
military training environment without causing changes in 
basic routines. 



1 



147 



141 



LIST OF REFERENCES 

Avner, R. A., & Misselt, A. L. Summary and critical analysis 
of USAOC&S CBE project evaluation. In H. A. Himwich (Ed.), 
. Summary and critical analysis of Aberdeen CBE project . 
Urbana, 111.: University of Illinois, Computer-based 
Education Research LaboVatory, 1977. ' 

Call-Himwich, E. Analysis of Aberdeen courseware: In H. A. 
Himwich (Ed.), Summary and Analysis of Aberdeen CBE proj ect 
Urbana, 111.: University of Illinois, Computer-based 
Education Research Laboratory, 1977. 

Campbell, D. T. , & Stanley', J. c. Experimental and quasi- 
experimental designs for research. In N. L. Gage (Ed.), 
- -iiandbook of research on teaching . Chicago: Rand McNally, 
1963. 

Cochran, Vf. G. Errors of measurement in statistics. Techno- 
metrics. 1968, ^0{^) , 637-666. . • 

Cronbach, L. j. Course improvement through evaluation. 
Teachers College Record , 1963, 6£, IO3-IO9. 

Dallman, B. E., DeLeo, P. J., Main, P. S., & Gillman, D. C. 
Evaluation of PLATO IV in vehicle maintenance training . 
(AFHRL -TP- 77-59) Lowry Air Force Base, Colorado 80230: 
Air Force Human Resources Laboratory, Technical Training 
division, 1977. 

Department bf the Army. Evaluation of the PLATO IV syste m in 
a military training environment . Aberdeen Proving Grounds, 
MD: U. S. Army Ordnance Center and School, 1975. 

Francis, L. Guidelines for es taolishing and managing a 

' ' u 

computer-based education site . Urbana, 111.: University 
of Illinois, Computer-based Education Research Laboratory, 
1977b. 



148 



142 



Francis, L. PLATC IV terminal peripheral devices , Urbana, 
111.: University of Illinois, Computer-based Education 
^- Research Laboratory, 1976a. ' 

Francis, L. • The TUTOR training course; Lessons learned . 

Urbana, 111.: University of Illinois, Computer-based 

Education Research Laboratory , ^ 1 976b . 
Francis, L. A comparison of the costs for illustrations 

pre:3ented- by the plasma panel and microfiche . Urbana, 

111.: University of Illinois, Computer-based Education 

Kesekrch Laboratory, 1977a. 
Green, J. 'S. One^year interim report on the AFHRL-ATC PLATO IV 
^ service test: 1972-1973 - (TR CH 73-622) Chanute Air 

Force Base, 111. 61868: USAF School of Applied Aerospace 

Sciences, Training Research Applications Branch, March 1973. 
Himwich, H. A. The Chanute AFB PLATO service test: Site 

history and management. In H. A. Himwich (Ed.), 

Critique and summary of the Chanute AFB CPE project . 

Urbana, 111.: University of Illinois, Computer-bas^d 

Education Research Laboratory, 1977a. Y 
Himwich, H. A. Some aspects of project establishment and 

management. In H. A. Himwich (Ed.), Summary and analysis 
^ of Aberdeen CBF Project . Urbana, 111.: University of 

Illinois, Computer-based Education Research Laboratory, 

1977b. . 

Kirk, R. E. Experimental design: Procedures for the behavioral 

sciences . Belmont, CA,: Brooks/Cole, 1968. 
Klecka, J. A. An overview of chanute lessons . Urbana, 111.: 

University of Illinois, Computer-based Education Research 

Laboratory, 1977a . 
Klecka, J. A. Three aspects of PLATO use at Chanute A FB.' 

Urbana,. 111.: University of Illinois, Computer-based 

Education Research Laboratory, 1977b. 



ERiC 



149 



o 143 



Klecka, J. A. Military instructor, attitudes toward computer 
based technical training. In A. L. Misselt (Ed.\ User 
attitudes toward CDE in a military technical training 
setting. Urbana, 111.: University of Illinois, Computer 
based Education Research Laboratory, 1977c. 

Lyman, E. R. PLATO highlights (5th Ed.). Urbana, 111.: 

University of Illinois, Computer-based Education Research 
Laboratory, 1977. 

Misselt, A. L. The effects of measurement error on analysis 
of covarl ance with multiple- covariates . Unpublished 
doctoral dissertation. University of Illinois, 1977. 

Popham, W. J. Criterion referenced measurement . Englewood 
Cliffs, N. J.: Educational Technology Publications, Inc. 
1971. ' . 

Salomon, G. & Clark, R. E. Reexamining the methodology of 
research on media and technology in education. Review of 
Educational Research . 197"/, £7, 99-120.' 

Seidel, R. J. Learner control of instructional sequenci ng 
within an adaptive tutorial CAI environment (HumRRO Tech. 
Rep. 75-7). Alexandria, Va.: Human Resources Research ' 
' Organization, June, 1975. 

Smith, S., i Sherwood, B. A. Educational uses of the PLATO 
computer system. Science, April 23, 1976, pp. 3'l'l-352. 

Steinberg, E. R. 'Review of student control in computer- 
assisted instruction. J ournal of Computer Based Instruc- 
tion , in press. / 

Tatsuoka, K., Misselt, A. L., & Maritz, P. Attitudes and 
performance of military students in computer-based 
technical training. In A. L. Misselt (Ed.), User atti- 
tudes toward CDE in a military technical training setting . 
Urbana, 111.: University of Illinois, Computer-based 
Education Research Laboratory, 1977. 



150 



ERIC 



Tatsuoka, K. Approaches to validation of criterion refer - 
e nced tests and computer-based instruction in a military 

project . UrbanaTlll.: University of Illinois, Computer 
based Education Research Laboratory, 1977. 

Tatsuoka, M. M. Multivariate analysis: Techniques foredu - 

^ cational and psychological research . New York: Wiley, 
1971. 



1/ 



\ 




